ETL: support organization id based reporting

soxofaan commented 9 months ago

Discussed with @JanssenBrm: we want to introduce an "organization" concept in the ETL API resource reporting, so that users that are part of multiple organizations/projects can explicitly specify the desired organization that resource usage should be reported under.

user specifies organization_id as job option (well known feature of batch jobs, also possible for sync jobs) during job creation time
organisation_id should be included in ETL resource reporting request (not supported yet)
ideally, organization_id should be checked for validity at job creation time. ETL endpoint to be added

jdries commented 9 months ago

and why not using the 'plan' option for this? That's part of the openEO API, so easier to standardize?

soxofaan commented 9 months ago

and why not using the 'plan' option for this?

In our discussion, @JanssenBrm mentioned also another thing on the roadmap that feels more suited for the "plan" field: distinction between different kind of credits: free, commercial, ..., . The "organization" concept matches a lot worse with how "plan" is currently described in the spec, e.g. orgs will be specific per user, while plans are expected to be declared globally in the capabilities doc.

soxofaan commented 6 months ago

this ticket is currently blocked: this feature first has to be implemented in ETL API : https://jira.vito.be/browse/MKTP-379

rahmandawibowo-vito commented 5 months ago

Hi @soxofaan, I'm picking up the ETL-API implementation for this ticket https://jira.vito.be/browse/MKTP-379. I'm just want to confirm the changes made on ETL-API to make this working:

The OpenEO needs to check if the job can be executed based on org ID using the new endpoint (link https://etl-dev.terrascope.be/docs/#/organizations/OrganizationsController_getOrganizationPermissions).
If the OpenEO sends an org ID on the reporting request (sync job or batch job), the ETL-API will try to deduct credits based on the org ID (even though user ID is also present on the payload). The deduction will be a success only if the user is the member of the org. The org ID should be the same with the one on the 1st point.

This also implies that whenever the organization based reporting is being used by OpenEO, these two points needs to be strictly followed to ensure a correct flow. This is because mixing up between the user and organization permission request can lead to unintended effect (e.g. the job can be executed on VITO org, but when deducting credits will be using other org that doesn't have sufficient credits).

Do you perhaps have some comments on this?

JeroenVerstraelen commented 4 months ago

@soxofaan is this issue still blocked?

soxofaan commented 4 months ago

hi @rahmandawibowo-vito ,

apologies for the slow response, the notification of this apparently slipped under my radar

The OpenEO needs to check if the job can be executed based on org ID using the new endpoint

Does this endpoint just check if a user is within the org, or does it also check if there is minimum amount of credits?

If the OpenEO sends an org ID on the reporting request (sync job or batch job), the ETL-API will try to deduct credits based on the org ID (even though user ID is also present on the payload). The deduction will be a success only if the user is the member of the org. The org ID should be the same with the one on the 1st point.

Is there a fallback mechanism when the user is not a member of the org, e.g. deduct the user's personal balance? Otherwise there is a loophole that can be exploited: start a heavy job while being member, and temporarily unregister as org member around the time the job finishes -> nobody is charged

e.g. the job can be executed on VITO org, but when deducting credits will be using other org that doesn't have sufficient credits

I'm not sure I understand why this is a problem specifically to the org-based reporting. You always have that problem, right, even with user-based reporting?

GET /organizations/orgId/permissions POST /resources > orgId

This orgId is apperently a number: how can the user figure out this id? Is that available somewhere on eoplaza?

soxofaan commented 1 week ago

@rahmandawibowo-vito this ticket got a priority bump on our end, so I'd like to pick this up again. From the questions in my previous comment, this is the most important one:

This orgId is apperently a number: how can the user figure out this id? Is that available somewhere on eoplaza?

rahmandawibowo-vito commented 1 week ago

Hi @soxofaan, yes the orgId information should be provided by the eoplaza. But I just realized that we don't display this info yet to the user. I think I will add this orgId somewhere in the eoplaza so that we can make a guideline/docs on this.

Open-EO / openeo-geopyspark-driver

ETL: support organization id based reporting #671