search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.89k stars 792 forks source link

Max retries exceeded with url: /v3/packages/platformio/tool/tool-scons #4341

Closed torntrousers closed 2 years ago

torntrousers commented 2 years ago

We've a platformio build that runs every hour and has started failing since 4:19pm on Jun 30th:

I don;t think we have changed anything, could it be something at the platformio end? Any ideas?

Here's a build log:

Running with gitlab-runner 14.5.2 (e91107dd)
   on New nGxj884L
Preparing the "shell" executor
00:00
 Using Shell executor...
Preparing environment
00:00
 Running on cq-ecl0[2](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L2)...
Getting source from Git repository
00:06
 Fetching changes with git depth set to 50...
 Reinitialized existing Git repository in /home/gitlab-runner/builds/nGxj884L/0/firmware-development/esp[3](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L3)2_tc03_demo/.git/
 Checking out db6ff930 as master...
 Skipping Git submodules setup
Executing "step_script" stage of the job script
01:[4](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L4)5
 $ docker login -u "$QL_AZURE_CR_USERNAME" -p "$QL_AZURE_CR_PASSWORD" $QL_AZURE_CR_URL
 WARNING! Using --password via the CLI is insecure. Use --password-stdin.
 WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json.
 Configure a credential helper to remove this warning. See
 https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 Login Succeeded
 $ docker build -f ./deployments/Dockerfile-test -t esp32-tc03-test-image-$CI_COMMIT_SHORT_SHA .
 Step 1/5 : FROM quarklink.azurecr.io/basebuild_ebed:pio_v1.0
  ---> a379a3bd7[6](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L6)eb
 Step 2/5 : RUN apt-get install openssh-client
  ---> Using cache
  ---> 4d[7](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L7)c58360f84
 Step 3/5 : WORKDIR /app
  ---> Using cache
  ---> 3c72b40785ba
 Step 4/5 : COPY . .
  ---> bdc[9](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L9)f3b644d9
 Step 5/5 : RUN chmod +x /app/deployments/build-artifact.sh
  ---> Running in 9aa62f658305
 Removing intermediate container 9aa62f658305
  ---> 00b77fb894ed
 Successfully built 00b77fb894ed
 Successfully tagged esp32-tc03-test-image-db6ff930:latest
 $ docker run -v /dev:/dev --privileged --name esp32-tc03-test-container-$CI_COMMIT_SHORT_SHA esp32-tc03-test-image-$CI_COMMIT_SHORT_SHA sh -c "platformio test --upload-port /dev/ttyUSB0 --test-port /dev/ttyUSB0"
 Verbose mode can be enabled via `-v, --verbose` option
 Collected 1 items
 Processing * in heltec_wifi_kit_32 environment
 --------------------------------------------------------------------------------
 Building...
 Tool Manager: Installing platformio/tool-scons @ ~4.40200.0
 Error: HTTPSConnectionPool(host='api.registry.ns1.platformio.org', port=443): Max retries exceeded with url: /v3/packages/platformio/tool/tool-scons (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1f68a2adc0>, 'Connection to api.registry.ns1.platformio.org timed out. (connect timeout=[10](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/40453#L10))'))
ivankravets commented 2 years ago

It looks like our system totally blocked your IP due to the thousands of requests to the registry. Did you cache ~/.platformio between each docker run?

torntrousers commented 2 years ago

Thousands of requests over what sort of time period?

ivankravets commented 2 years ago

Could you share the first and last octet of the public machine's IP?

ivankravets commented 2 years ago

Do you use PlatformIO Core 6.0? It caches duplicated requests internally avoiding DDoS-type attacks.

ivankravets commented 2 years ago

Please reopen if you still need help

gsmith246 commented 2 years ago

Please would you unblock IP 81.19.60.132 for us.

Many thanks Greg

ivankravets commented 2 years ago

This IP is not blocked.

gsmith246 commented 2 years ago

Un yet we still get :

Building... Tool Manager: Installing platformio/tool-scons @ ~4.40200.0 Error: HTTPSConnectionPool(host='api.registry.ns1.platformio.org', port=443): Max retries exceeded with url: /v3/packages/platformio/tool/tool-scons (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7fa261ef9b50>, 'Connection to api.registry.ns1.platformio.org timed out. (connect timeout=10)'))

ivankravets commented 2 years ago

You need to check your IP directly from the machine where PlatformIO works. It seems IPs are different.

gsmith246 commented 2 years ago

Thanks ... working on it

gsmith246 commented 2 years ago

Hi Ivan

Can please try and un block ip address 88.151.155.60

Thanks

Best Regards

Greg

[CQ_logo]https://video.wixstatic.com/video/68379b_8883894bdbb649708251ed48b1806b27/720p/mp4/file.mp4

Greg Product Engineering Manager

m: +44 7967 388967<tel:+447827820250> e: https://www.cryptoquantique.com/ @.**@.> w: www.cryptoquantique.comhttps://www.cryptoquantique.com/

Crypto Quantique Limited | London @.***,-0.1025479,17z/data=!3m1!4b1!4m5!3m4!1s0x487605189bfa826b:0x60678a50bc1e1a7d!8m2!3d51.5038438!4d-0.1003592?hl=en%20>

Connect with us:LinkedInhttps://www.linkedin.com/company/crypto-quantique/

From: Ivan Kravets @.> Sent: 11 July 2022 10:29 To: platformio/platformio-core @.> Cc: Greg Smith @.>; Manual @.> Subject: [EXTERNAL]: Re: [platformio/platformio-core] Max retries exceeded with url: /v3/packages/platformio/tool/tool-scons (Issue #4341)

CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

You need to check your IP directly from the machine where PlatformIO works. It seems IPs are different.

— Reply to this email directly, view it on GitHubhttps://github.com/platformio/platformio-core/issues/4341#issuecomment-1180170927, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AEIRPKPUDM3HX4OKAQ4HTVTPSN7ANCNFSM52QN4X4Q. You are receiving this because you are subscribed to this thread.Message ID: @.***>

ivankravets commented 2 years ago

Done! Please check that you use the latest PlatformIO 6.1.1 or above and map the container's ~/.platformio folder to the external folder on the host machine. Run a few times docker container and ensure that you don't see package redownloading on each "build".

gsmith246 commented 2 years ago

Working again,

Thank you for your help and support.

Best Regards

[CQ_logo]https://video.wixstatic.com/video/68379b_8883894bdbb649708251ed48b1806b27/720p/mp4/file.mp4

Greg Product Engineering Manager

m: +44 7967 388967<tel:+447827820250> e: https://www.cryptoquantique.com/ @.**@.> w: www.cryptoquantique.comhttps://www.cryptoquantique.com/

Crypto Quantique Limited | London @.***,-0.1025479,17z/data=!3m1!4b1!4m5!3m4!1s0x487605189bfa826b:0x60678a50bc1e1a7d!8m2!3d51.5038438!4d-0.1003592?hl=en%20>

Connect with us:LinkedInhttps://www.linkedin.com/company/crypto-quantique/

From: Ivan Kravets @.> Sent: 11 July 2022 12:25 To: platformio/platformio-core @.> Cc: Greg Smith @.>; Manual @.> Subject: [EXTERNAL]: Re: [platformio/platformio-core] Max retries exceeded with url: /v3/packages/platformio/tool/tool-scons (Issue #4341)

CAUTION:This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Done! Please check that you use the latest PlatformIO 6.1.1 or above and map the container's ~/.platformio folder to the external folder on the host machine. Run a few times docker container and ensure that you don't see package redownloading on each "build".

— Reply to this email directly, view it on GitHubhttps://github.com/platformio/platformio-core/issues/4341#issuecomment-1180280495, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2AEIRNFQXSNGDHDNJF4KULVTQABHANCNFSM52QN4X4Q. You are receiving this because you are subscribed to this thread.Message ID: @.***>

ivankravets commented 2 years ago

Hi @gsmith246, we still experience multiple duplicated requests from your IP. Have you resolved the issue with mapping ~/.platformio to a separate volume?

See https://docs.docker.com/storage/volumes/

Log in to the Docker instance and type pio system info. You will see that path to the PlatformIO Core. Also, the download logs show you use outdated PIO Core 5.2.3 that does not support caching for duplicated requests/packages. Please upgrade to the latest PlatformIO Core 6.

gsmith246 commented 2 years ago

We are trying to build a Dockerfile with all the packages we need for the project. However we are have issues with the tools/scans from your selfs, as you can see below, we have tried to install it but it will not install inside the container image. Whats the best way to install this package inside a Docker image? If we need a simple starter app? got any working examples of how best to do this within Docker? Thanks,


#12 0.607
#12 0.607 WARNING: This command is deprecated and will be removed in the next releases.
#12 0.607 Please use `pio pkg install` instead.
#12 0.607
#12 0.609 Platform Manager: Installing platformio/tool-scons
#12 1.306 Error: Could not find the package with 'platformio/tool-scons' requirements for your system 'linux_x86_64'```
ivankravets commented 2 years ago

Yes, this is great to have a pre-built docker image with all PlatformIO packages.

WARNING: This command is deprecated and will

What was that command?

What do you need to make PlatformIO work even in the offline mode without any dependencies on the external packages:

  1. Override libdeps-dir with something ~/.platformio/libdeps
  2. Keep ~/.platformio in the container.

Please check other folders documented in https://docs.platformio.org/en/latest/projectconf/section_platformio.html You can override them with environment variables.

Which folders are important for you?

  1. core_dir (PlatformIO installs here dependent packages, toolchains, and dev-platforms)
  2. libdeps_dir - PlatformIO installs here project-dependent libraries.

Did it help you?

torntrousers commented 2 years ago

The command is:

RUN pio platform install platformio/tool-scons

which fails with:

Error: Could not find the package with 'platformio/tool-scons' requirements for your system 'linux_x86_64'
ivankravets commented 2 years ago

You need to pass a type of package. See https://docs.platformio.org/en/latest/core/userguide/pkg/cmd_install.html

Otherwise, we don't know the type of package. There is some info in the registry but we will fail with git/zip/tar.gz sources. That was the reason why we asked a developer to specify a type of package manually.

RUN pio platform install -t platformio/tool-scons

But! You don't need to install packages one by one. Just navigate to your project and type pio pkg install. It will install ALL dependencies.

torntrousers commented 2 years ago

Even after a pio pkg install it still doesn't seem to pre-install it and it gets downloaded with each build - here is a build log, see on line 90:

Building...
[70](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L70) Library Manager: Installing olikraus/U8g2 @ 2.28.8
[71](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L71) Unpacking
[72](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L72) Library Manager: U8g2@2.28.8 has been installed!
[73](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L73) Library Manager: Installing paulstoffregen/OneWire @ 2.3.5
[74](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L74) Unpacking
[75](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L75) Library Manager: OneWire@2.3.5 has been installed!
[76](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L76) Library Manager: Installing milesburton/DallasTemperature @ 3.9.1
[77](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L77) Unpacking
[78](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L78) Library Manager: DallasTemperature@3.9.1 has been installed!
[79](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L79) Library Manager: Resolving dependencies...
[80](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L80) Library Manager: Installing knolleary/PubSubClient @ 2.8.0
[81](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L81) Unpacking
[82](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L82) Library Manager: PubSubClient@2.8.0 has been installed!
[83](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L83) Library Manager: Installing m5stack/M5Core2 @ 0.1.0
[84](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L84) Unpacking
[85](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L85) Library Manager: M5Core2@0.1.0 has been installed!
[86](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L86) Library Manager: Installing throwtheswitch/Unity @ ^2.5.2
[87](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L87) Unpacking
[88](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L88) Library Manager: Unity@2.5.2 has been installed!
[89](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L89) Tool Manager: Installing platformio/tool-scons @ ~4.40300.0
[90](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L90) Downloading
[91](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L91) Unpacking
[92](https://dev-gitlab.cryptoquantique.com/firmware-development/esp32_tc03_demo/-/jobs/42017#L92) Tool Manager: tool-scons@4.40300.220706 has been installed!
ivankravets commented 2 years ago

Tool Manager: tool-scons@4.40300.220706 has been installed!

I see. It is an internal package. I think we can install it too. Please open a separate feature request https://github.com/platformio/platformio-core/issues

The workaround is pio pkg install -t platformio/tool-scons

Library Manager:

Have you overridden libdeps_dir or workspace_dir? The working environment dependencies are installed into the isolated location. You need to keep it in a container too.

Why you don't want to clone your PIO project and run pio run? It will download ALL packages required for the build process, including tool-scons, etc. You will just need to cache core_dir + libdeps_dir. You can control the location of these dirs. The compiled objects go to the build_dir, which is workspace_dir/build == .pio/build in the root of project.

Summary:

ENV PLATFORMIO_CORE_DIR=/platformio
ENV PLATFORMIO_LIBDEPS_DIR=/platformio_libdeps
RUN cd some/pio/project
RUN pio run

Does it make sense?

torntrousers commented 2 years ago 
pio platform install -t platformio/tool-scons
Usage: pio platform install [OPTIONS] [PLATFORM...]
Try 'pio platform install -h' for help.

Error: No such option: -t

C:\Users\tornt>pio --version
PlatformIO Core, version 6.1.3
ivankravets commented 2 years ago

Sorry, it was my typo. “pio pkg install…”

ivankravets commented 2 years ago

@torntrousers, do you need any help with caching project library dependencies (libdeps_dir)?