search

reacherhq / backend

⚙️ REST backend for Reacher - Open-Source Email Verification API.
https://reacher.email
Other
218 stars 60 forks source link

Getting "io: incomplete " #231

Closed Zonalds closed 2 years ago

Zonalds commented 3 years ago

Hey,

I am exploring Reacher and would get this:

"smtp": { "error": { "type": "SmtpError", "message": "io: incomplete" } },

Any idea what can be done to fix this.

I am running on docker, have port 25 open, and using a SOCK proxy in the request.

amaury1093 commented 3 years ago

Hey @Zonalds ! For these errors, the best way is to have a look at logs. You need to set the RUST_LOG env variable to debug in your Docker container, and then run docker logs.

If you paste the redacted logs here, I can try to look into it.

Zonalds commented 3 years ago

Hi @amaurym. Thank you very much. Here is the log:

[2021-10-06T01:25:11Z INFO reacher_backend::sentry_util] Running Reacher v0.3.9 [2021-10-06T01:25:11Z INFO heroku] Server is listening on 0.0.0.0:8080. [2021-10-06T01:25:11Z INFO warp::server] Server::run; addr=0.0.0.0:8080 [2021-10-06T01:25:11Z INFO warp::server] listening on http://0.0.0.0:8080 [2021-10-06T01:26:23Z DEBUG hyper::proto::h1::io] parsed 7 headers [2021-10-06T01:26:23Z DEBUG hyper::proto::h1::conn] incoming body is content-length (195 bytes) [2021-10-06T01:26:23Z DEBUG hyper::proto::h1::conn] incoming body completed [2021-10-06T01:26:23Z DEBUG reacher_backend::routes::check_email::post] [email=support@zoho.com] Checking with retry option Direct, attempt #2 [2021-10-06T01:26:23Z DEBUG check_if_email_exists] Checking email "support@zoho.com" [2021-10-06T01:26:23Z DEBUG check_if_email_exists] Found the following syntax validation: SyntaxDetails { address: Some(EmailAddress("support@zoho.com")), domain: "zoho.com", is_valid_syntax: true, username: "support" } [2021-10-06T01:26:23Z DEBUG trust_dns_proto::xfer::dns_handle] querying: zoho.com MX [2021-10-06T01:26:23Z DEBUG trust_dns_resolver::name_server::name_server_pool] sending request: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:23Z DEBUG trust_dns_resolver::name_server::name_server] reconnecting: NameServerConfig { socket_addr: 8.8.8.8:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: true } [2021-10-06T01:26:23Z DEBUG trust_dns_proto::xfer] enqueueing message: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:23Z DEBUG trust_dns_resolver::name_server::name_server] reconnecting: NameServerConfig { socket_addr: 8.8.4.4:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: true } [2021-10-06T01:26:23Z DEBUG trust_dns_proto::xfer] enqueueing message: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:23Z DEBUG trust_dns_proto::udp::udp_stream] created socket successfully [2021-10-06T01:26:23Z DEBUG trust_dns_proto::udp::udp_stream] created socket successfully [2021-10-06T01:26:23Z DEBUG trust_dns_proto::udp::udp_client_stream] received message id: 34941 [2021-10-06T01:26:23Z DEBUG trust_dns_proto::xfer::dns_exchange] io_stream is done, shutting down [2021-10-06T01:26:23Z DEBUG trust_dns_proto::xfer::dns_exchange] io_stream is done, shutting down [2021-10-06T01:26:23Z DEBUG check_if_email_exists] Found the following MX hosts MxDetails { lookup: Ok(MxLookup(Lookup { query: Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }, records: [Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 473, rdata: MX(MX { preference: 50, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 51, 122, 111, 104, 111, 99, 111, 109], label_ends: [7, 11, 14] } }) }, Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 473, rdata: MX(MX { preference: 20, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 50, 122, 111, 104, 111, 99, 111, 109], label_ends: [7, 11, 14] } }) }, Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 473, rdata: MX(MX { preference: 10, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 122, 111, 104, 111, 99, 111, 109], label_ends: [6, 10, 13] } }) }], valid_until: Instant { tv_sec: 49736, tv_nsec: 759672181 } })) } [2021-10-06T01:26:23Z DEBUG check_if_email_exists] Found the following misc details: MiscDetails { is_disposable: false, is_role_account: true } [2021-10-06T01:26:23Z DEBUG check_if_email_exists::smtp] Connecting to smtpin3.zoho.com.:25 [2021-10-06T01:26:23Z INFO fast_socks5::client] Connected @ 216.185.35.115:51766 [2021-10-06T01:26:23Z DEBUG fast_socks5::client] Send version and method len [5, 1] [2021-10-06T01:26:23Z DEBUG fast_socks5::client] client auth methods supported: [0] [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Socks version (5), method chosen: 0. [2021-10-06T01:26:24Z INFO fast_socks5::client] No auth will be used [2021-10-06T01:26:24Z INFO fast_socks5::client] Requesting headersSome(Domain("smtpin3.zoho.com.", 25))... [2021-10-06T01:26:24Z DEBUG fast_socks5::client] TargetAddr::Domain [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Bytes long version: [5, 1, 0, 3, 17, 115, 109, 116, 112, 105, 110, 51, 46, 122, 111, 104, 111, 46, 99, 111, 109, 46, 0, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Bytes shorted version: [5, 1, 0, 3, 17, 115, 109, 116, 112, 105, 110, 51, 46, 122, 111, 104, 111, 46, 99, 111, 109, 46, 0, 25] [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Padding: 24 [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Reply received: [version: 5, reply: 0, rsv: 0, address_type: 1] [2021-10-06T01:26:24Z DEBUG fast_socks5::util::target_addr] Address typeIPv4 [2021-10-06T01:26:24Z INFO fast_socks5::client] Remote server connected to 216.185.35.115:51766. [2021-10-06T01:26:24Z DEBUG check_if_email_exists::smtp] Closing smtpin3.zoho.com.:25, because of error 'io: incomplete'. [2021-10-06T01:26:24Z DEBUG async_smtp::smtp::client::inner] >> QUIT<CRLF> [2021-10-06T01:26:24Z DEBUG reacher_backend::routes::check_email::post] [email=support@zoho.com] Got result with retry option Direct, attempt #2, is_reachable=Unknown [2021-10-06T01:26:24Z DEBUG reacher_backend::routes::check_email::known_errors] Io error: incomplete [2021-10-06T01:26:24Z DEBUG reacher_backend::routes::check_email::post] [email=support@zoho.com] Checking with retry option Direct, attempt #1 [2021-10-06T01:26:24Z DEBUG check_if_email_exists] Checking email "support@zoho.com" [2021-10-06T01:26:24Z DEBUG check_if_email_exists] Found the following syntax validation: SyntaxDetails { address: Some(EmailAddress("support@zoho.com")), domain: "zoho.com", is_valid_syntax: true, username: "support" } [2021-10-06T01:26:24Z DEBUG trust_dns_proto::xfer::dns_handle] querying: zoho.com MX [2021-10-06T01:26:24Z DEBUG trust_dns_resolver::name_server::name_server_pool] sending request: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:24Z DEBUG trust_dns_resolver::name_server::name_server] reconnecting: NameServerConfig { socket_addr: 8.8.8.8:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: true } [2021-10-06T01:26:24Z DEBUG trust_dns_proto::xfer] enqueueing message: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:24Z DEBUG trust_dns_resolver::name_server::name_server] reconnecting: NameServerConfig { socket_addr: 8.8.4.4:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: true } [2021-10-06T01:26:24Z DEBUG trust_dns_proto::xfer] enqueueing message: [Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }] [2021-10-06T01:26:24Z DEBUG trust_dns_proto::udp::udp_stream] created socket successfully [2021-10-06T01:26:24Z DEBUG trust_dns_proto::udp::udp_stream] created socket successfully [2021-10-06T01:26:24Z DEBUG trust_dns_proto::udp::udp_client_stream] received message id: 47376 [2021-10-06T01:26:24Z DEBUG trust_dns_proto::xfer::dns_exchange] io_stream is done, shutting down [2021-10-06T01:26:24Z DEBUG trust_dns_proto::xfer::dns_exchange] io_stream is done, shutting down [2021-10-06T01:26:24Z DEBUG check_if_email_exists] Found the following MX hosts MxDetails { lookup: Ok(MxLookup(Lookup { query: Query { name: Name { is_fqdn: false, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, query_type: MX, query_class: IN }, records: [Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 527, rdata: MX(MX { preference: 50, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 51, 122, 111, 104, 111, 99, 111, 109], label_ends: [7, 11, 14] } }) }, Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 527, rdata: MX(MX { preference: 20, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 50, 122, 111, 104, 111, 99, 111, 109], label_ends: [7, 11, 14] } }) }, Record { name_labels: Name { is_fqdn: true, label_data: [122, 111, 104, 111, 99, 111, 109], label_ends: [4, 7] }, rr_type: MX, dns_class: IN, ttl: 527, rdata: MX(MX { preference: 10, exchange: Name { is_fqdn: true, label_data: [115, 109, 116, 112, 105, 110, 122, 111, 104, 111, 99, 111, 109], label_ends: [6, 10, 13] } }) }], valid_until: Instant { tv_sec: 49791, tv_nsec: 633170760 } })) } [2021-10-06T01:26:24Z DEBUG check_if_email_exists] Found the following misc details: MiscDetails { is_disposable: false, is_role_account: true } [2021-10-06T01:26:24Z DEBUG check_if_email_exists::smtp] Connecting to smtpin3.zoho.com.:25 [2021-10-06T01:26:24Z INFO fast_socks5::client] Connected @ 216.185.35.115:51766 [2021-10-06T01:26:24Z DEBUG fast_socks5::client] Send version and method len [5, 1] [2021-10-06T01:26:24Z DEBUG fast_socks5::client] client auth methods supported: [0] [2021-10-06T01:26:25Z DEBUG fast_socks5::client] Socks version (5), method chosen: 0. [2021-10-06T01:26:25Z INFO fast_socks5::client] No auth will be used [2021-10-06T01:26:25Z INFO fast_socks5::client] Requesting headersSome(Domain("smtpin3.zoho.com.", 25))... [2021-10-06T01:26:25Z DEBUG fast_socks5::client] TargetAddr::Domain [2021-10-06T01:26:25Z DEBUG fast_socks5::client] Bytes long version: [5, 1, 0, 3, 17, 115, 109, 116, 112, 105, 110, 51, 46, 122, 111, 104, 111, 46, 99, 111, 109, 46, 0, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] [2021-10-06T01:26:25Z DEBUG fast_socks5::client] Bytes shorted version: [5, 1, 0, 3, 17, 115, 109, 116, 112, 105, 110, 51, 46, 122, 111, 104, 111, 46, 99, 111, 109, 46, 0, 25] [2021-10-06T01:26:25Z DEBUG fast_socks5::client] Padding: 24 [2021-10-06T01:26:25Z DEBUG fast_socks5::client] Reply received: [version: 5, reply: 0, rsv: 0, address_type: 1] [2021-10-06T01:26:25Z DEBUG fast_socks5::util::target_addr] Address typeIPv4` [2021-10-06T01:26:25Z INFO fast_socks5::client] Remote server connected to 216.185.35.115:51766. [2021-10-06T01:26:25Z DEBUG check_if_email_exists::smtp] Closing smtpin3.zoho.com.:25, because of error 'io: incomplete'. [2021-10-06T01:26:25Z DEBUG async_smtp::smtp::client::inner] >> QUIT [2021-10-06T01:26:25Z DEBUG reacher_backend::routes::check_email::post] [email=support@zoho.com] Got result with retry option Direct, attempt #1, is_reachable=Unknown [2021-10-06T01:26:25Z DEBUG reacher_backend::routes::check_email::known_errors] Io error: incomplete [2021-10-06T01:26:25Z INFO reacher_backend::sentry_util] Sending info to Sentry: is_reachable=Unknown [2021-10-06T01:26:25Z INFO warp::filters::log] 172.17.0.1:51434 "POST /v0/check_email HTTP/1.1" 200 "-" "Thunder Client (https://www.thunderclient.io)" 1.724690175s [2021-10-06T01:26:25Z DEBUG hyper::proto::h1::io] flushed 483 bytes [2021-10-06T01:26:25Z DEBUG hyper::proto::h1::conn] read eof

`

I tried the last 3 docker images on dockerhub, same error. Let me know if you need anything else.

Kasalop commented 3 years ago

I encounter the same issue. My best guess here would be that the target e-mail server is blocking the connection because my system is not a real e-mail server. I think there is an dns entry missing or something like that.

Zonalds commented 3 years ago

Curious, were you using a proxy?

I would find it has to do with the proxy, not the mail server. We changed proxy and everything works fine now.

Kasalop commented 3 years ago

Hi @Zonalds, no proxy was used. Plain new debian 10. I installed docker, run the container, called the api => io: incomplete for some email domains (not for all, gmail works, gmx does not work).

Zonalds commented 3 years ago

Yep, the issue would come back we decide to just use an in-house script instead. If you find a solution, I would love to know.

Kasalop commented 3 years ago

I got it working so far, @Zonalds. @Amaurym provided me a link to another issue that suggested to check the smtp connection to the smtp servers by hand using telnet. I did that and got the real smtp error message instead of the io: incomplete error message. The io incomplete message seems to be a by-product produced by the software because it tries to send or receive something from the smtp connection that was already closed by the other smtp server. Sadly, it does not point to the real issue here.

With the real smtp eror message I had to do different things for different ESPs to make them accept connections from my server / the reacher application:

In summery: configure a valid e-mail server without a mta but with reacher installed.

z3d3m0n commented 2 years ago

I got it working so far, @Zonalds. @amaurym provided me a link to another issue that suggested to check the smtp connection to the smtp servers by hand using telnet. I did that and got the real smtp error message instead of the io: incomplete error message. The io incomplete message seems to be a by-product produced by the software because it tries to send or receive something from the smtp connection that was already closed by the other smtp server. Sadly, it does not point to the real issue here.

With the real smtp eror message I had to do different things for different ESPs to make them accept connections from my server / the reacher application:

  • make sure PTR Record of the used IP is valid
  • make sure MX Records for the used hostname were set
  • SPF Record for the domainname included server ip
  • get the used ip address removed from blocklists

In summery: configure a valid e-mail server without a mta but with reacher installed.

Hello @Zonalds , having the same issue , tested , ss , 3proxy and dante what proxy provide are you using ?

amaury1093 commented 2 years ago

I'm closing this. My personal tests (which I can share privately, not putting it public as they contain emails) show that with the recent upgrade to async-smtp 0.5.0, these io: incomplete errors are way less frequent.

If someone still sees them with check-if-email-exists >= v0.8.32, please ping here.