search

balena-io / open-balena

Open source software to manage connected IoT devices at scale
https://balena.io/open
GNU Affero General Public License v3.0
1.03k stars 162 forks source link

Failed to get device type build data for... #176

Closed shaunco closed 10 months ago

shaunco commented 10 months ago

Attempting to start an open-balena instance results in open-balena-api having hundreds of errors during startup that like:

Failed to get device type build data for imx8mmebcrs08a1/2.103.3 Access Denied AccessDenied: Access Denied
    at Request.extractError (/usr/src/app/node_modules/aws-sdk/lib/services/s3.js:711:35)
    at Request.callListeners (/usr/src/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/usr/src/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/src/app/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/usr/src/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/src/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/src/app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/src/app/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/src/app/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/usr/src/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
    at Request.emit (/usr/src/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/usr/src/app/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/usr/src/app/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/src/app/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/src/app/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/src/app/node_modules/aws-sdk/lib/request.js:38:9)

As detailed in https://github.com/balena-io/balena-cli/issues/2322 and https://github.com/balena-io/open-balena/issues/129 by @pdcastro, open-balena points to balena-cloud's S3 bucket full of endpoint images.

At startup, open-belena-api grabs https://resin-production-img-cloudformation.s3.amazonaws.com/ to get a list of all available device types - this succeeds. open-balena-api then attempts to fetch device-type.json under each key. S3 returns AccessDenied for every one of those requests when IMAGE_STORAGE_PREFIX is set to images - which is the default at https://github.com/balena-io/open-balena/blob/ea07d43c5c7e98bb8b11c808b12428a46019c8d5/compose/services.yml#L34

If I change IMAGE_STORAGE_PREFIX to esr-images, then open-balena-api starts up without all the access denied errors ... but I'm not sure if that is giving me the right set of images or not.

shaunco commented 10 months ago

For anyone else following along, esr-images contains the "Extended Support Release" images as detailed here and is a very limited set of devices:

... so, esr-images is not what I wanted, as it is missing the 3 device types I actually need 😅

As for the errors, there seem to be certain device types where everything has permissions set to not allow public access for reading files, but still allows listing files. For example, the error above is trying to read https://resin-production-img-cloudformation.s3.amazonaws.com/images/imx8mmebcrs08a1/2.100.3/device-type.json which gives access denied (as does every other file in images/imx8mmebcrs08a1/*), this causes contractSync to be able to enumerate the files but not fetch the device-type.json and thus the open-balena-api log gets flooded with these errors on first sync.

Seems like there are two possible fixes:

  1. Fix resin-production-img-cloudformation permissions on these device types to either prohibit both LIST and GET, or permit GET.
  2. Create a new ENV variable that can be set for open-balena-api that allows for a list of device-types we actually want. For example: INCLUDE_IMAGES=iot-gate-imx8,generic-amd64,generic-aarch64 which could filter both the device type contracts pulled from https://github.com/balena-io/contracts and the device type information pulled from https://resin-production-img-cloudformation.s3.amazonaws.com/images/ ... as I would assume no open-balena instance actually needs every possible device type - and if they do, they can just not set INCLUDE_IMAGES and it would fall back to the current behavior of grabbing everything.
shaunco commented 10 months ago

Closing in favor of https://github.com/balena-io/open-balena-api/issues/1433