search

Open-EO / openeo-geopyspark-driver

OpenEO driver for GeoPySpark (Geotrellis)
Apache License 2.0
25 stars 4 forks source link

openeo-dev.vgt.vito.be logs to https://etl.terrascope.be #495

Open bossie opened 10 months ago

bossie commented 10 months ago

@JanssenBrm: web app and job_tracker should log to https://etl-dev.terrascope.be instead.

openeo-dev_to_etl_prod

job_tracker-dev_to_etl_prod

bossie commented 10 months ago

We need to support different ETL APIs but the problem is that on Terrascope, the application will fetch its ETL API credentials from a certain secret in Vault, of which the path is hard-coded.

One option is to make this path configurable; another option is to eliminate the application's runtime dependency on Vault altogether and let the infrastructure inject the credentials by means of environment variables. This is exactly what happens on k8s for ETL where Terraform interacts with Vault and also on Terrascope for EJR where Jenkins takes care of this.

A downside of this approach is that the application will have to be restarted if the credentials are compromised and have to change; an upside, on the other hand, is that there's no longer a runtime dependency on Vault so a Vault failure has no immediate consequences.

Also: https://github.com/Open-EO/openeo-geopyspark-driver/blob/540181e933ac6cab87d0b74c038186686281d07f/openeogeotrellis/backend.py#L1482-L1485

bossie commented 10 months ago

As discussed: not extremely urgent.

jdries commented 10 months ago

@JanssenBrm what about prod users using our dev instance?

bossie commented 4 months ago

@JanssenBrm does this still apply (cfr. the remark by @jdries)?

JanssenBrm commented 3 months ago

This still applicable. However, the fix is not very trivial in the sense that the ETL API only supports a single authentication system for its endpoints. Specifically for openEO, the production etl (etl.terrascope.be) is linked with the production TerraScope keycloak (sso.terrascope.be) and development (etl-dev.terrascope.be) with the staging TerraScope keycloak. So that means, if you link openeo-dev to etl-dev, you'll get issues with production users that use the dev instance.

Right now the problem is more related to the reporting (I guess). So I'm wondering if we can just split the reporting using the sourceId in the resources request on openEO. This is a free text field and it could be populated with a different value for dev and production. This way we can already make the split on the reporting side of things.