PCAPI /fs/ changes to optimize for Survey IDs

[xmichael]

Now that PCAPI is queried directly for Surveys we need to optimize per-survey querying and grouping of information. This is mostly for keeping things in order - - a database will still be used for indexing.

This change affects only PCAPI local provider:

Implementation:

• Identify FTOpen and Survey Design /fs/ calls and switch to higher level -- as long as they use /records/, /editors/,/features/ instead of low level /fs/ calls they are backwards compatible. A new github branch should be created just in case to avoid breaking stuff. This is a necessary change independently of this issue
• Change PCAPI high-level (e.g. /records/) to use the new scheme underneath: i.e '/records/local/UUID/Name' used to translate to '/fs/local/UUID/records/NAME/records.json' but now it should translate to '/fs/local/UUID/SID/records/NAME/record.json instead.

NOTE: while this issue more for avoiding namespace colitions it brings many speed optimizations for free. It specificatlly optimizes the following queries:

• Return all observatinons made to a public survey
• Return all observations of an individual (private or public) from all surveys
• Return all observations of a participant from a specific survey.

It does not optimize returning all observations pertaining to a private survey though. This is too cobweb specific and is indexed via database to allow for fluid requirements.

[panterz]

Example for editors:

/fs/local/UUID/SID/editors/SID.edtr
/fs/local/UUID/SID/editors/image.jpg
/fs/local/UUID/SID/editors/decision-tree.json

or

/fs/local/UUID/SID/editors/index.edtr
/fs/local/UUID/SID/editors/image.jpg
/fs/local/UUID/SID/editors/decision-tree.json

or

/fs/local/UUID/editors/SID.edtr
/fs/local/UUID/editors/SID/image.jpg
/fs/local/UUID/editors/SID/decision-tree.json

FTGB transfer:

Before

editors/<editor1>.edtr
editors/<editor2>.edtr
editors/<editor3>.edtr
records/<record>/record.json

After

<editor1>/editors/<editor1>.edtr
<editor2>/editors/<editor2>.edtr
<editor3>/editors/<editor3>.edtr
<editor>/records/<record>/record.json

[panterz]

I have an example of a form that includes images and decision tree with images. At the moment the structure on the pcapi is:

editors/
        5106d3aa-99ac-4186-b50d-6fcfdf9946f4.edtr
        <img>.[jpg,png]
        5106d3aa-99ac-4186-b50d-6fcfdf9946f4-1.zip

where the images are included inside the form and the zip file contains the decision tree with its images.

Ideally I would like this:

editors/
         5106d3aa-99ac-4186-b50d-6fcfdf9946f4.edtr
         5106d3aa-99ac-4186-b50d-6fcfdf9946f4/
                                              5106d3aa-99ac-4186-b50d-6fcfdf9946f4-1.zip
                                              <img>.[jpg,png]

The reason I cannot apply it now is because the FTOpen is download all these through the editors interface of pcapi that cannot go inside subfolder, it looks only on the directory level of editors.

[xmichael]

This is issue currently has proposals for both /records/ and /editors/

It should be noted that:

• /editors/ changes are minimal and backwards compatible as described in the previous post. It is useful for keeping DTs/Assets in different folders.
• /records/ changes are more urgent and are expected to give 10-fold speedup and clearer datasets. However they require extensive changes in FTOpen. First we need a list of all possible /records/ requests from FTOpen and see if they need to pass the extra SID parameter in the new version.