Do not rely on cloud.3mdeb.com to download test data

macpijan commented 11 months ago

The problem you're addressing (if any)

We should not rely on cloud.3mdeb.com to download test data:

it increases cloud.3mdeb.com usage, which is not necessary,
in case of outage, tests would not work,
cannot be contributed to by external contributors.

Describe the solution you'd like

Use a repo independent of 3mdeb internal infrastructure, which we can still control and approve external contributions. Most likely in a form of a github repository.

Where is the value to a user, and who might that user be?

No response

Describe alternatives you've considered

No response

Additional context

This comes from this discussion

But this is not the only place where we rely on the cloud.3mdeb.com:

[ ] lib directory

./esp-scanning-lib.robot:${SHIMX64_URL}=         https://cloud.3mdeb.com/index.php/s/TipPHgsCj6MCZHi/download/shimx64.efi
./esp-scanning-lib.robot:${GRUBX64_URL}=         https://cloud.3mdeb.com/index.php/s/e2JjYANjMsgyXtS/download/grubx64.efi
./esp-scanning-lib.robot:${DISK_IMAGE_URL}=      https://cloud.3mdeb.com/index.php/s/BwLyjHT9fRncXMY/download/image.img
./secure-boot-lib.robot:${GOOD_KEYS_URL}=           https://cloud.3mdeb.com/index.php/s/SjM5dQGji4XDAni/download/good_keys.img
./secure-boot-lib.robot:${NOT_SIGNED_URL}=          https://cloud.3mdeb.com/index.php/s/zmJXxGG4piGB2Me/download/not_signed.img
./secure-boot-lib.robot:${BAD_KEYS_URL}=            https://cloud.3mdeb.com/index.php/s/BJPbSqRH6NdbRym/download/bad_keys.img
./secure-boot-lib.robot:${BAD_FORMAT_URL}=          https://cloud.3mdeb.com/index.php/s/AsBnATiHTZQ6jae/download/bad_format.img

[ ] dasharo-compatibility directory

./logo-customization-functionality.robot:    Download File    https://cloud.3mdeb.com/index.php/s/rsjCdz4wSNesLio/download    /tmp/logo.bmp

[ ] keywords.robot file

        Download File    https://cloud.3mdeb.com/index.php/s/C6LJMi4bWz3wzR9/download    ${cbmem_path}
        Download File    https://cloud.3mdeb.com/index.php/s/D7AQDdRZmQFTL6n/download    ${flashrom_path}
        Download File    https://cloud.3mdeb.com/index.php/s/ScCf8XFLZYWBE25/download    ${cbfstool_path}

macpijan commented 3 months ago

@BeataZdunczyk We need to update the status of where cloud is used. Then, we need to propose alternative solution. Even creating a git repo as a submodule with test files, can be a viable option.

macpijan commented 2 months ago

@wiktormowinski Let's start with updating a list of required components.

Maybe the new repo makes sesnse, as it was one of the ideas in linked discussion. This repo then would be a submodule of OSFV.

Maybe we can enable LFS in this repo if there will be many binary files there.

wiktormowinski commented 2 months ago

@macpijan Updated list looks like that keywords.robot:

Download File    https://cloud.3mdeb.com/index.php/s/C6LJMi4bWz3wzR9/download    ${cbmem_path}
Download File    https://cloud.3mdeb.com/index.php/s/fsPNM8SpDjATMrW/download    ${flashrom_path}
Download File    https://cloud.3mdeb.com/index.php/s/ScCf8XFLZYWBE25/download    ${cbfstool_path}

dasharo-compatibility/dcu.robot:

    Download To Host Cache
    ...    logo.bmp
    ...    https://cloud.3mdeb.com/index.php/s/rsjCdz4wSNesLio/download
    ...    6e5a6722955e4f78d947654630f27ff833703fbc04776ffed963c96617f6bb2a

in lib/esp-scannning-lib.robot

 ${DISK_IMAGE_URL}=              https://cloud.3mdeb.com/index.php/s/BwLyjHT9fRncXMY/download/image.img

lib/secure-boot-lib.robot

${GOOD_KEYS_URL}=           https://cloud.3mdeb.com/index.php/s/SjM5dQGji4XDAni/download/good_keys.img
${NOT_SIGNED_URL}=          https://cloud.3mdeb.com/index.php/s/zmJXxGG4piGB2Me/download/not_signed.img
${BAD_KEYS_URL}=            https://cloud.3mdeb.com/index.php/s/BJPbSqRH6NdbRym/download/bad_keys.img
${BAD_FORMAT_URL}=          https://cloud.3mdeb.com/index.php/s/AsBnATiHTZQ6jae/download/bad_format.img

macpijan commented 2 months ago

Let's try with new repo for test data, that would contain test files, and ideally also some documentation/scripts how to prepare them.

On the other hand, if we have script to produce these binary files, storing them in binary form in repo might be redundant. Maybe this should be second step, and here we would just focus on dropping cloud entirely, and on modyfing test scripts to use files from this new repo instead.

wiktormowinski commented 2 months ago

sure, any requests regarding the name of this new submodule (and other such parameters)?

macpijan commented 2 months ago

No. osfv-test-data?

Dasharo / open-source-firmware-validation