Manage credentials

[sebastian-luna-valero]

Hi @lorincmeszaros

As discussed in https://github.com/c-scale-community/use-case-hisea/issues/13#issuecomment-954703322 we need to find a way to safely work with credentials on GitHub, and I am opening this ticket to track process on that.

[sebastian-luna-valero]

Hi,

I have been looking for ways to deal with this issue and there is no direct solution. Let's then agree on the best way forward.

After googling, I found the following:

• GitHub has encrypted secrets which are available for GitHub Actions, which is currently out of the scope of what we are trying to solve here.
• Similar thing to GitHub Codespaces which is not applicable either.
• Here you are a good summary of the state of the art in secret management.

My preferred approach going forward would be something simpler (admittedly with its own drawbacks):

1. Let's exchange credentials using the infrastructure allocated to the Use Case. For HiSea we could have a shared folder inside the VM in the GRNET cloud.
2. We use .gitignore to leave credential files untracked, and adapt the scripts to treat this accordingly (e.g. in https://github.com/c-scale-community/use-case-hisea/issues/18 we could update the download_era5.py to check whether the expected ${HOME}/.cdsapi file exists before doing anything else, and when the file is not there, print out a warning asking the user to request the credentials via c-scale-wp4@mailing-list.eodc.eu)

Could you please let me know your thoughts?

Many thanks, Sebastian

[lorincmeszaros]

Hi Sebastian,

I agree with your proposed solution.

@avgils since you were adopting the download_era5.py, can we do what Sebastian proposed? This solution would also apply to the CMEMS download scripts.

[lorincmeszaros]

We were re-discussing this with @avgils. Due to the terms described in https://github.com/c-scale-community/use-case-hisea/issues/18 it would be safer if new users actually create their credentials and CDS API keys. In this way we are not liable for misuse of our scripts. Creating a new user and the CDS API key is actually simple as documented here. I will still contact ECMWF and ask them if an EU project can be considered an entity/organiation or if every user should create this separately.

Followingthe safer case the script could be adjusted to check whether${HOME}/.cdsapi is there and if not, print a warning asking the user create account and API keys.

@backeb what do you think?

[sebastian-luna-valero]

Hi @lorincmeszaros @avgils

The credentials won't be shipped with the code. We would create the credentials for C-SCALE but only share it within the consortium in order to automate/reproduce results. This way external users would need to create their own and we avoid the issue, right?

Many thanks, Sebastian

[backeb]

Hi guys,

I think how we deal with the credentials depends a bit on the feedback we get from ECMWF on question:

if an EU project can be considered an entity/organiation or if every user should create this separately.

I think we should certainly do the check:

if ${HOME}/.cdsapi is there and if not, print a warning asking the user create account and API keys

Regarding:

We would create the credentials for C-SCALE but only share it within the consortium in order to automate/reproduce results.

Is the point here so that the project can reproduce what the use cases have done? I think this is fine, unless you want to assign project participants as "testers", e.g. @sebastian-luna-valero and @bschumac can be testers of the workflows that we implement. But either approach is fine by (i.e. a project credential or individual user credentials)

We should certainly somewhere, perhaps in the scripts' readme file, make some sort of disclaimer statement about Deltares not being held liable for how these scripts are used...

[sebastian-luna-valero]

Hi,

Sure, let's wait for ECMWF's answer.

Regarding C-SCALE credentials, I was thinking both from the testing point of view, and from the re-usability point of view inside the consortium (e.g. something developed by a use case can be reused by a different use case). Not sure whether I am simply making things up! :) In that case, I guess C-SCALE should be liable, and not a single partner (again, if that's applicable).

Best regards, Sebastian

[sebastian-luna-valero]

Hi again,

By the way, in this meeting with the LSDA Use Case, they were considering the use of HashiCorp Vault as a way to manage credentials. A joint investigation may be the best way to solve this.

Best regards, Sebastian

[lorincmeszaros]

Hi all,

Update on the response of ECMWF (see attached conversation):

**_Registering one single account and sharing credentials within the Consortium in not a problem if this simplify your internal way of working.

However one single user account shared might generate some bottle necks as our QualityOfService rules which control and limit accesses (eg. number of concurrent request per user) would apply at the level of the shared UID.

The more accounts you create, the resources you can allocate in parallel.

Suggestion is then to improve performance when accessing the system to create more than one account with some criteria in the name-organization that wil allow us to link the origin of the users (eg. consortium name as organization) for statistical purposes._**

CAMS and C3S Support - Service Desk.pdf

[sebastian-luna-valero]

Thanks @lorincmeszaros

Good to know that we can share one single account. Remember that the initial goal was to just re-run the download_era5.py script and avoid having:

Exception: Client has not agreed to the required terms and conditions.. To access this resource, you first need to accept the termsof 'Licence to use Copernicus Products' at https://cds.climate.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products

I think next steps should be discussed at WP4 level and agree on a common approach going forward. I shared my personal opinion on how to deal with this in: https://github.com/c-scale-community/use-case-hisea/issues/17#issuecomment-967023191

[backeb]

Thanks @sebastian-luna-valero and @lorincmeszaros for both of you input.

I think we should follow the approach you suggest in https://github.com/c-scale-community/use-case-hisea/issues/17#issuecomment-967023191

We could think of use cases where different approaches might be needed and document these somewhere, e.g. in confluence.

[backeb]

We have added instructions on how to create CMEMS and CDS user accounts.