search

ClusterCockpit / cc-backend

Web frontend and API backend server for ClusterCockpit Monitoring Framework
https://www.clustercockpit.org
MIT License
14 stars 12 forks source link

Make full JobArchive available per simple API call #203

Closed aw32 closed 4 months ago

aw32 commented 11 months ago

At the moment one needs to do multiple API calls to reconstruct the information of the JobArchive or get the JobArchive from the file system of the backend server. It would be nice to have one API call to receive the complete JobArchive.

moebiusband73 commented 10 months ago

Sorry for the late reply.

Do you mean to get the job archive as a regular user or as admin getting the complete archive? The job archive can get pretty big, and I guess it is not feasible to download usin a API.

We plan to provide a S3 based backend for the job archive. This would then allow to access or download the job archive from a network location independent of cc-backend. Would this be a solution to your problem?

aw32 commented 10 months ago

Do you mean to get the job archive as a regular user or as admin getting the complete archive? The job archive can get pretty big, and I guess it is not feasible to download usin a API.

There is already a REST API that is accessible with an API token. With "complete archive" I mean the JobArchive files (meta.json and data.json) for one job as it is stored on disk. Right now the REST API only allows to get certain metrics for a job, but that is quite cumbersome.

We plan to provide a S3 based backend for the job archive. This would then allow to access or download the job archive from a network location independent of cc-backend. Would this be a solution to your problem?

We already can access the files by using an additional method (e.g. network filesystem, ssh, etc.). This issue is about extending the REST API with a simple and easy way.

moebiusband73 commented 10 months ago

Ok, sorry for the misunderstanding. What we could do to send the complete information that is contained in both meta.json and data.json as one large json (with keys meta and data). Or do you need the files specifically?

Another option would be a flag to also include all metrics and extend the current job schema with the metric data. What do you think? Or do you have another suggestion?

aw32 commented 10 months ago

I think the first suggestion (one large json with data and meta key) would be the easiest. If necessary, the json can be easily split into the two original json files.

aw32 commented 10 months ago

It would also be useful to get the cluster specifications for the available clusters. In case of a change, one could easily get the current one using the API. Edit: To be more specific: this could be a separate API call that simply returns an array with all the cluster specifications.

moebiusband73 commented 10 months ago

Ok, that sound doable. I see if we can do it for the next release.