search

recast-hep / recast-flask-frontend

attempt at a new frontend for RECAST
0 stars 2 forks source link

persistent file storage (ZENODO storage + DOI minting for requests) #17

Open lukasheinrich opened 8 years ago

lukasheinrich commented 8 years ago

we need some way to store the recast requests archives on the server, so that they can be retrieved later. so for this we need some fixed filestructure.

In heroku this can be tricky, but this can work:

https://devcenter.heroku.com/articles/s3

lukasheinrich commented 8 years ago

another idea is using GitHub as storage, but not sure how difficult it is to sync back to server

lukasheinrich commented 8 years ago

@cbora, actually probably the nicest and easiest way would be it we would just hook up the request submission directly to ZENODO. In the control center we already have the response side hooked up to ZENODO and it would create a nice symmetry between request and response (also fixed DOIs for both)

see code example here: https://github.com/recast-hep/recast-control-center-prototype/blob/master/recastcontrolcenter/backendtasks.py#L95

and actual API calls here: https://github.com/recast-hep/recast-control-center-prototype/blob/master/recastcontrolcenter/backendtasks.py#L30

cbora commented 8 years ago

@lukasheinrich OK, I will switch to ZENODO if you think that's better. Though, I had already integrated AWS S3. Currently, I am looking into file naming strategy to avoid filenames collisions, and so far I think using uuid is the best solution. If you have a better one let me know.

lukasheinrich commented 8 years ago

we might still need local storage, since I'm not sure that the only storage we ever need is the request data, so it's good to have something additional to zenodo. Maybe the upload to zenodo can be done as an additional step after it was copied to amazon z3. Also, if we upload to Zenodo, we probably should have DOI fields in the database model for BasicRequests / PointRequests), should we have a new issue for that?

As for filenaming.. at least the request data should come with their unique request uuid (scan_request_id field / point_request_id). can't we use this to build a folder structure.