Live Paper related KG Schema and API issues

[appukuttan-shailesh]

List of items to be resolved:

• [ ] Corresponding author email field missing (GDPR issues)

• [ ] Created author email field missing (GDPR issues)

• [ ] Approved author email field missing (GDPR issues)

  (Note: in future it might be possible to show email addresses of EBRAINS account holders)

• [x] Allow different people for corresponding author and approval author - approval author wil be set as custodian

• [x] Add support for resource items from the KG (e.g. models, traces, morphologies)

• [x] Add missing fields: version, date modified.... - _replace created_date with modifieddate

• [x] Avoid duplication check by article title (paper_title). Currently multiple entries with same title cannot be made. Might be required during dev phase to have multiple variants of LP before finalizing. We should rather add a field name or alias.

• [ ] PUT end-point should preferably return the LP object (as opposed to null); in accordance with other V2 VF PUT API end-points

• [ ] PUT end-point returning code 200 (success) even for non-existent UUIDs

• [x] PUT end-point creating new LP entries when asked to update existing LP's name (paper title), even though UUID of LP to be updated is specified in invoked URL.

• [ ] POST end-point updates entry (acts as PUT) for requests with existing paper titles; not raising the Exception as intended in code. [Awaited]

• [ ] POST example: https://validation-v2.brainsimulation.eu/docs#/Live%20Papers/create_live_paper_livepapers__post "id" field should be removed from example for POST (?)

• [x] For LivePaper, "url" and "doi" should be allowed to be empty (string/null) and not checked for URL format (during dev phase, it is useful to be able to create placeholders with empty urls) - empty string not allowed for "url" and "doi", null allowed

• [ ] For LivePaperResourceItem, "url" and "view_url" ("mc_url") should be allowed to be empty (string/null) and not checked for URL format (during dev phase, it is useful to be able to create placeholders with empty urls) _download url ("url") is mandatory, "mcurl" is optional; all empty strings to be replaced by null

• [ ] Internal Server Error - meanignless data, but cause unknown (i.e. no useful debug info for user as to issue with payload). (see example)

• [ ] With POST, occasionally getting error: Code: Undocumented. TypeError: Failed to fetch. Couldn't find any associated log records on server. Same query seems to work when fired immediately again. - will wait to see if this issue resurfaces

• [ ] Add a live paper visibility field: private, public, password-protected (only public are considered published live papers and listed on the web page, and so we need a method to filter by status of live papers) - add new field to schema to keep track of status [Awaited]

• [x] How to add a password field (to password protect document for review phase)... or should be handled via Collab membership? Will necessitate reviewers to have EBRAINS accounts! - plans to add a password field; store value with hashing+salt - implemented with password field; not using Collab membership

• [ ] for custom resource sections, can dataFormatted be mapped to description within the API layer, such that the payload POSTed will continue to use dataFormatted. I am keen to maintain the same terminology for fields across different sections within the tool, where in other places description field denotes something else. If not possible at the API layer, I will add a method to reformat the payload when dealing with KG. - mapping will be done on API layer (to sumarize.... 'data' field can now be removed [see last point below] and API layer will map "dataFormatted" field in payload (produced by LP tool) to "description" field in KG Schema; has to be done done for GET, PUT and POST) [Awaited]

• [ ] discuss how to prevent strict checks on input data while saving to KG in pre-publish model (e.g. about having a LivePaperDraft schema with less strict checks) - Andrew will think and decide [Awaited / To be discussed]

• [ ] is Collab based access permissions implemented for GET, PUT, POST Live Paper REST API end-points? - Yes, access permissions to be implemented

• [ ] DELETE end-point in Live Paper APIs - lower priority

• [ ] Discuss about naming live_paper_title vs live_paper_name [Awaited / To be discussed]

• [ ] GET, POST, (PUT?) return response with key as title ... I suppose this should be updated to return both live_paper_title and associated_paper_title

• [ ] REST APIs return version and lp_tool_version fields. LP tool uses only 'lp_tool_version' and this should be mapped onto version. Currently 'lp_tool_version' returns a fixed value of "0.1" (seemingly; as value doesn't get updated when somethign else specified) and 'version' returns null value. [Awaited]

• [ ] LP tool has been updated such that it no longer requires data field, as it directly works on dataFormatted. The former can now be deleted.

• [ ] POST causing INTERNAL SERVER ERROR when resource url is set to null. [Awaited]

• [ ] Problem with how live paper resource items are saved on KG. I believe their URL is used as the unique key This is preventing the possibility to set labels when the same item is used in different live papers. Problem exists for both KG and non-KG entries - probably the best solution is to change the KG representation so that it uses both identifier and section as the unique key - [Awaited]

• [ ] When trying to save LPs with KG entries I am getting "500 Internal Server Error" (save problem resolved)

• [ ] API for electrophysiological recordings (see here) needs to have more filters setup. I can imagine having the following: 'species', 'cell_type', ['brain_region'?] Question: docs for this end-point shows 'brain_region' as a filter. How is this filter applied? I don't see any brain region related info in the retrieved JSON response. Info for 'species' and 'cell_type' is available under 'recorded_from' key for applicable entries. - [Awaited]

• [x] implement CORS proxy server ourselves: https://github.com/Rob--W/cors-anywhere

Note:

• Made one small fix and deployed on server: https://github.com/appukuttan-shailesh/hbp-validation-framework/commit/eb3d9d6e8d2e9b83d12f9d57c4e1214f8518ad53 and so had to change the git origin on the server to my fork of this repo

[apdavison]

• We should rather add a field name or alias.

I propose to rename paper_title to associated_paper_title and add live_paper_title rather than name or alias, to avoid ambiguity in which paper is being referred to. ok @appukuttan-shailesh ?

[appukuttan-shailesh]

That's a good idea. Lets go with that @apdavison. But how about using live_paper_name just so that it isn't expected to match the "title" header displayed on the live paper page?

[apdavison]

That's something to discuss. I think the live paper page should display its own name/title as the title, not the title of the associated paper/preprint.

[appukuttan-shailesh]

Some issues that need more immediate attention: 1) Password verification issue for protected live papers 2) Issue with abstract & license not retrieved via GET 3) Implement alias for live paper direct URL 4) Handle 'publication' of live papers via KG mechanism; update REST APIs accordingly 5) Authentication seem to time out too fast (~20-30 mins); can handle via refresh tokens?

[appukuttan-shailesh]

@apdavison .... a reminder that "license" still isn't being returned by the GET end-points. Receiving null.