search

jasonwhite / rudolfs

A high-performance, caching Git LFS server with an AWS S3 and local storage back-end.
MIT License
392 stars 39 forks source link

Connection reset by peer #57

Open nebnes opened 1 year ago

nebnes commented 1 year ago

I encounter an error when I am running the docker-compose.minio.yml locally : after a git push

batch response: Post "http://localhost:8081/api/nebnes/dirty-lfs-test/objects/batch": read tcp 127.0.0.1:38150->127.0.0.1:8081: read: connection reset by peer
error: failed to push some refs to 'github.com:nebnes/dirty-lfs-test.git

I have a test repo with this .lsconfig : 

[lfs]
url = "http://localhost:8081/api/nebnes/dirty-lfs-test"

Thanks in advance

jasonwhite commented 1 year ago

Do you have any logs from the server? This error indicates that the server has crashed and the connection was dropped.

nebnes commented 1 year ago

I don't think it is crashing. Here is the container log: 

2023-03-07T08:26:34.173Z INFO  rudolfs > Initializing storage...
2023-03-07T08:26:34.173Z INFO  rudolfs::storage::s3 > Connecting to S3 bucket 'test' at region 'localhost'
solareon commented 1 year ago

Running into the same issue. If you wait approximately 13 minutes or so it finally times out and spits out an error as follows.

 2023-09-29T17:48:00.325Z INFO  rudolfs > Initializing storage...
 2023-09-29T17:48:00.325Z INFO  rudolfs::storage::s3 > Connecting to S3 bucket 'kickenitrp-lfs' at region 'us-west-1'
 2023-09-29T18:01:43.851Z ERROR rudolfs              > Error during dispatch: error trying to connect: Unsupported scheme http

I have verified that the minio is working with the minio client and created a bucket named kickenitrp-lfs and can place and read objects from it without issue. I haven't tried wrapping the minio with https to see if that works yet or not.

Edit: Figured it out. There was a trailing / on the AWS_S3_ENDPOINT that was making it's way into the request to the minio server that broke it.

Edit2: So it appears that the issue is the port reference in the AWS_S3_ENDPOINT. When I had it pointed to a cloudflare tunnel it worked without any issues due to no port number in the env var. Once I redid the config to have it point locally due to the minio and rudolfs being on the same box no need to send the traffic to cloudflare to come back that caused it to break as well.

jasonwhite commented 1 year ago

@solareon Thanks for digging into this. It sounds like this is a configuration issue? It's not super clear to me what should be fixed on the Rudolfs side of things, whether it be documentation or code. Did minio stop supporting non-HTTPS connections?

For starters, I think one fix is to avoid retrying the connection in this particular situation so it fails much faster. However, it looks like Rusoto is returning an HttpDispatch error, which may include transient errors like timeouts.

solareon commented 1 year ago

@jasonwhite Not a problem at all. Almost all of the other git lfs impementations seem to be abandoned or unmaintained and the git-lfs-test server while it works ain't really all that great. From looking at it further it appears to be a code issue. When a port is defined in the AWS_S3_ENDPOINT it looks like it just gets hung up and infinitely retries and hitting the backoff code before finally erroring out.

Minio doesn't seem to have any issues with the connection being http or https as my test with https was with a cloudflare tunnel pointed at the http endpoint.

solareon commented 1 year ago

Looking at this more I think that the underlying issue is the rusoto has deprecated connecting via http and requires the connection to be over https. I tried doing some hostname mapping for the container and connecting over http and it didn't even try to connect.

jasonwhite commented 1 year ago

Indeed, that looks like the problem: https://github.com/rusoto/rusoto/issues/1980 (With a potential workaround.)

Rudolfs should probably migrate away from Rusoto, since it is now unmaintained. aws-sdk-s3 might be a good replacement, but it says isn't production ready.