search

tizbac / proxmoxbackupclient_go

In development proxmox backup client written in golang, aimed at windows compatibility
GNU General Public License v3.0
193 stars 14 forks source link

TASK ERROR: connection error: timed out - several hours #34

Open Gaetann18 opened 2 months ago

Gaetann18 commented 2 months ago

Good morning,

first of all I would like to thank you for the work provided because this software saves me a lot of manipulation to save my files for my remote clients!

However, I encounter a random error on large backup volumes (2TB), after several hours a timeout appears without necessarily a valid reason and this stops the entire process. Would you be able to handle this exception and put a retry?

I thank you in advance !

2024-08-06T02:01:38+00:00: upload_chunk done: 2021309 bytes, 22163b19ac75767c38c1c347bb012eea5c917d1458f67b6b04e1a0b888f9dbe9 2024-08-06T02:01:38+00:00: POST /dynamic_chunk 2024-08-06T02:01:38+00:00: upload_chunk done: 4336542 bytes, 3672b23732968a0eb082caaeaaac9dabdd4d5c366fd9e1365c26b1f75d163d3a 2024-08-06T02:01:38+00:00: POST /dynamic_chunk 2024-08-06T02:01:39+00:00: upload_chunk done: 4145846 bytes, b2d66ca9ce089429431d5dc4de4fedf1c4614c321e8e19a9b006bf648aef3a41 2024-08-06T02:01:39+00:00: POST /dynamic_chunk 2024-08-06T02:15:21+00:00: backup failed: connection error: timed out 2024-08-06T02:15:21+00:00: removing failed backup 2024-08-06T02:15:21+00:00: TASK ERROR: connection error: timed out 2024-08-06T02:15:21+00:00: POST /dynamic_chunk: 400 Bad Request: error reading a body from connection: timed out

tizbac commented 2 months ago

We could add a retry indeed but it looks like a connectivity issue to me, maybe some cable flapping, also recovering from broken http2 connection may not be doable at all

Il mar 6 ago 2024, 08:13 Gaetann @.***> ha scritto:

Good morning,

first of all I would like to thank you for the work provided because this software saves me a lot of manipulation to save my files for my remote clients!

However, I encounter a random error on large backup volumes (2TB), after several hours a timeout appears without necessarily a valid reason and this stops the entire process. Would you be able to handle this exception and put a retry?

I thank you in advance !

2024-08-06T02:01:38+00:00: upload_chunk done: 2021309 bytes, 22163b19ac75767c38c1c347bb012eea5c917d1458f67b6b04e1a0b888f9dbe9 2024-08-06T02:01:38+00:00: POST /dynamic_chunk 2024-08-06T02:01:38+00:00: upload_chunk done: 4336542 bytes, 3672b23732968a0eb082caaeaaac9dabdd4d5c366fd9e1365c26b1f75d163d3a 2024-08-06T02:01:38+00:00: POST /dynamic_chunk 2024-08-06T02:01:39+00:00: upload_chunk done: 4145846 bytes, b2d66ca9ce089429431d5dc4de4fedf1c4614c321e8e19a9b006bf648aef3a41 2024-08-06T02:01:39+00:00: POST /dynamic_chunk 2024-08-06T02:15:21+00:00: backup failed: connection error: timed out 2024-08-06T02:15:21+00:00: removing failed backup 2024-08-06T02:15:21+00:00: TASK ERROR: connection error: timed out 2024-08-06T02:15:21+00:00: POST /dynamic_chunk: 400 Bad Request: error reading a body from connection: timed out

— Reply to this email directly, view it on GitHub https://github.com/tizbac/proxmoxbackupclient_go/issues/34, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBV6XZADELRRFO7RGK3S3ZQBSPHAVCNFSM6AAAAABMBTJYP6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TAMBTGAZTSMI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

Gaetann18 commented 2 months ago

may be the result of my load balancer which manages several fibers to balance network loads.

In all cases there is no network interruption being in high availability. And I've never had the problem before. I'm also going to investigate on my own but this is strange

tizbac commented 2 months ago

Http2 lives with a single tcp connection so I'm pretty sure it won't be recoverable , do a try with netcat and some random data to see if you get broken connection too , I guess yes

Il mar 6 ago 2024, 09:07 Gaetann @.***> ha scritto:

may be the result of my load balancer which manages several fibers to balance network loads.

In all cases there is no network interruption being in high availability. And I've never had the problem before. I'm also going to investigate on my own but this is strange

— Reply to this email directly, view it on GitHub https://github.com/tizbac/proxmoxbackupclient_go/issues/34#issuecomment-2270539076, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBV6UNGPAKQRPKNWCNS6LZQBY2DAVCNFSM6AAAAABMBTJYP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZQGUZTSMBXGY . You are receiving this because you commented.Message ID: @.***>

tizbac commented 2 months ago

Can you update me if it was a network issue? thanks

Gaetann18 commented 4 weeks ago

Good morning,

Unfortunately this is not a network problem. This happens even with a stable connection (I recently re-tested that’s why it took me a while to respond)

tizbac commented 4 weeks ago

Do you have AV software running? Can you try disabling it? Can also you try building it on Linux and reproduce there?

Il ven 4 ott 2024, 11:55 Gaetann @.***> ha scritto:

Good morning,

Unfortunately this is not a network problem. This happens even with a stable connection (I recently re-tested that’s why it took me a while to respond)

— Reply to this email directly, view it on GitHub https://github.com/tizbac/proxmoxbackupclient_go/issues/34#issuecomment-2393319159, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABBV6WNVW24PX3YVAMKT3TZZZQXZAVCNFSM6AAAAABMBTJYP6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJTGMYTSMJVHE . You are receiving this because you commented.Message ID: @.***>