Running in docker gives EOF error on PUT request

[ckortekaas]

Firstly - thanks for this extremely useful addon for Turborepo! Very impressive!

I have no problems using the downloadable executable, but using the docker container from the registry or built locally gives this error when turbo tries to connect eg:

2021-12-28T16:58:10.774+1000 [DEBUG] run: performing request: method=PUT url=http://127.0.0.1:8080/v8/artifacts/dad54db4bd311da2?teamId=team_test
2021-12-28T16:58:10.807+1000 [ERROR] run: request failed: error="Put "http://127.0.0.1:8080/v8/artifacts/dad54db4bd311da2?teamId=team_test": EOF" method=PUT url=http://127.0.0.1:8080/v8/artifacts/dad54db4bd311da2?teamId=team_test

That's running with an apiUrl and teamId set in ./turbo/config.json via turbo eg:

TURBO_TOKEN=testtoken yarn turbo run build  -vvv --force

Other things I've tried result in a connection refused error instead, so EOF seems to imply it's connecting but failing to PUT. I'm using S3 with valid credentials that work with the non-docker version. I've even modified the container to execute inside it manually various commands to make sure it's not that the command arguments are not being mistranslated somehow.

This is what I've been using in docker-compose:

  tapico-turborepo-cache:
    container_name: dev_tapico-turborepo-cache
    image: ghcr.io/tapico/tapico-turborepo-remote-cache:sha-152ba94
    networks:
      - internal
    ports:
      - "8080:8080"
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
    entrypoint: /bin/sh
    command: -c "/go/bin/tapico-turborepo-remote-cache --turbo-token=testtoken --kind=s3 --s3.region=us-east-1 --bucket=test-turborepo-cache -v"

The is on Turbo 1.0.23 and 0.0.8.

I'm a little rusty on my Docker and haven't used Go before but I couldn't see anything obviously wrong in either.

Is it possible to show a working example of a docker-compose wrapped around tapico-turborepo-remote-cache that allows the host (eg MacOS) to run turbo against it successfully? Thanks!

[weyert]

Thank you for reporting. I will have a look at it tomorrow :)

I typically use the environment variable alternative for the flags were possible when running the docker image. I didn't see the EOF file before. I will try to reproduce the issue locally.

@ckortekaas I typically don't use AWS myself, do you happen to know if Minio would be good enough or would I need a Amazon S3?

[ckortekaas]

Hi @weyert, this EOF happens even just using the local version, eg sh'ing into the container and running:

./tapico-turborepo-remote-cache --turbo-token=testtoken --kind=local -v

It shows:

# ./tapico-turborepo-remote-cache --turbo-token=testtoken --kind=local -v
projectID:  kind: local localStoragePath:  aws.endpoint:  google.endpoint:  google.credentialsJsonPath: Starting the Tapico Turborepo remote cache server

But EOF errors still show up when trying to connect. So something fundamental about container networking from a MacOS host and maybe something with the Go network listener?

[ckortekaas]

@weyert ok I've figured it out, this works running inside the container via a shell:

 LISTEN_ADDRESS=:8080 ./tapico-turborepo-remote-cache --turbo-token=testtoken --kind=local -v

That tells http.ListenAndServe to listen on any hostname, whereas at the moment the default is to only listen on 'localhost:8080'. Since calling a docker container from another container or from the host will usually make the call via a different hostname the Go library will never accept this I suspect.

So I guess the fix for this is maybe adding to the LISTEN_ADDRESS as a configuration flag, or defaulting to listening to just ':8080' (any host) instead of 'localhost:8080' on line 547 of main.go?:

        err := http.ListenAndServe("localhost:8080", loggedRouter)

[weyert]

Yeah, probably we should just listen to 0.0.0.0:8080 instead. I should update the README file with the latest state of things.

My apologies for the lack of documenting this LISTEN_ADDRESS.

[ckortekaas]

Awesome! BUCKET_NAME is another good env var to add to the docs too. AWS ECS Fargate as well as docker-compose were way easier to get the syntax right using environment variables. Thanks again for your inspiring project!

[weyert]

Awesome, glad you like the project :)

Don't hesitate to raise a PR for the docs or instructions how to get it running on AWS. That would be amazing!

[weyert]

@ckortekaas I have created an example and updated the README.md in the root. Would you be so kind and have a look at the two README files and the docker-compose example and whether you think it would be a useful example?

See: https://github.com/Tapico/tapico-turborepo-remote-cache/tree/main/examples/with-docker-compose

[tapico-weyert]

I also tried to reproduce the EOF issue of this ticket but I haven't been able to make it happen using the above mentioned docker-compose file. Could it be a AWS S3 specific thing? 🤔

[ckortekaas]

@weyert That looks great! I suspect the reason you might not have the issue with EOF is a difference in host OS. I am using macOS 12.1 with Docker Desktop 4.3.2 (the latest version). Are you perhaps on a different host OS and older or different version of Docker Desktop?

[weyert]

Ah I am on 4.3.1 and macOS 11. I am updating Docker now and will try again. Are you seeing the issue with the docker compose example?

[weyert]

I haven't had much luck yet to reproduce your problem. I will try to upgrade to macOS 12.1

[weyert]

@ckortekaas I have seen it occasionally earlier this week. I am wondering if it could be to do with some default maximum upload size in Golang for the PUT route? I will look into this

[weyert]

I am wondering if this ticket is related? https://github.com/vercel/turborepo/issues/487 Could the problem be on Turbo's side?