Create credentials for youtube scraper integration tests

benoit74 commented 3 months ago

As part of Dan GSOC, we will add few basic end-to-end integration tests to the CI so that we at least ensure the scraper is not completely broken, like we already have for zimit.

The distinction is that for youtube scraper we for now need to pass an API key to the scraper, and maybe also an optimization cache URL.

I propose to:

create a new youtube API key (with a new project) so that it is not breaking the production should the CI goes wild
- due to the dynamic nature of Github Actions shared runners, there will be no IP whitelisting on this API key but this key is not supposed to leak since it will be only a CI secret
create a dedicated wasabi user (and it keys) with custom policy allowing access only to the youtube dev bucket

benoit74 commented 3 months ago

@rgaudin any comment?

rgaudin commented 3 months ago

LGTM

benoit74 commented 3 months ago

Done:

in youtube I've failed to create a project on my own user but I achieved to create a project in kiwix.org organization (it was the contrary few months ago ... don't know what changed) ; details of the project are stored in the Google spreadsheet used by content team
in wasabi I've created a youtube-ci user, with a custom policy KiwixYoutubeCI allowing full S3 operations only on org-kiwix-dev-youtube bucket attached directly to the user

OPTIMIZATION_URL and YOUTUBE_API_KEY are configured in Github openzim/youtube CI secrets for proper operation of the scraper in the CI

kiwix / operations

Create credentials for youtube scraper integration tests #216