mrbotcr / py3ClimMob

ClimMob is software for agricultural citizen science
https://climmob.net/
GNU Affero General Public License v3.0
5 stars 2 forks source link

Explore ODK- Enketo Metadata feasibility. #312

Open MarManrow opened 1 month ago

MarManrow commented 1 month ago

Jacob has requested a change in the metadata format for ClimMob trials, suggesting that it should become an ODK (Open Data Kit) form instead of being an integrated module in the interface. He explained that this change would make the metadata more adaptable to various emerging project types, such as variety testing, consumer testing, or agronomic practices. He also highlighted that implementing this change would offer greater flexibility, allowing each project type to have a tailored format for capturing relevant information. To maintain the ODK metadata forms, he suggested that each thematic area have a "thematic coordinator" responsible for managing and updating ClimMob's metadata and libraries.

The MrBot development team will analyze this request and provide feedback.

BrandonMrBot commented 1 month ago

Incorporating Enketo for completing metadata forms for ClimMob projects requires developing endpoints and interfaces tailored to handle the metadata functionality. Several considerations must be addressed to achieve this:

The forms must be accessible without requiring a username and password, unlike the typical Enketo setup, where authentication is usually necessary.

It is not possible to use the predefined information in the project definition within the ODK form as it is generic for all projects and no default values can be set (e.g. auto-completion of the “project name” field).

Once metadata is submitted to ClimMob from Enketo, editing the data is not possible. The only solution is to manage data updates through ClimMob's interface or start a new form to replace the existing record.

The metadata must meet certain standardization criteria to ensure consistency across all projects, enabling platform super administrators to conduct effective analysis later on.

Every time the ODK form changes, processes must be executed to store the already registered metadata so that it can be reloaded after the update. This is necessary because the database needs to be reconstructed from scratch for proper functionality.

If the update involves changes in column names, structure types, or data types, it could lead to the loss of previously recorded metadata.

A method must be devised to relate the metadata registered in Enketo with a corresponding project in ClimMob.

Using ClimMob users for ODK’s selection lists does not work automatically, requiring procedures that demand time and resources for continuous updates.

Metadata updates based on feedback and progress (Form already implemented in ClimMob):

MarManrow commented 1 month ago

@jacobvanetten and @marieALaporte Take a look at Brandon´s reply.

jacobvanetten commented 1 month ago

Perhaps still think a bit further about ODK/Enketo.

But I am not pushing for Enketo per se. The comment was about Enketo, but the thought behind it was to make a more streamlined process from metadata standard to form/interface.

MarManrow commented 1 month ago

This issue resulted in an intermediate solution that displays the metadata forms in the interface, but they are composed of JSON template files. The team agreed on this solution during today's SCRUM, but @BrandonMrBot will provide more details. @marieALaporte might also offer feedback based on her experience.

jacobvanetten commented 1 month ago

@kauedesousa mentioned https://codemeta.github.io/ as a source of inspiration. For data, this builds on https://codemeta.github.io/crosswalk/datacite/ This would be good to look at, as it has some additional fields, like funder, which could be useful.

jacobvanetten commented 1 month ago

https://fairware.metadatacenter.org/ - this checks data if it is FAIR https://metadatacenter.org/ (launch workbench) - this helps users create metadata in JSON format based on ontologies

BrandonMrBot commented 1 month ago

In the previous SCRUM (Thursday 10-10-2024) I talked about a proposal for the implementation of metadata using JSON files, the following is the documentation of the possible flow that this functionality could have.

Link: https://docs.google.com/document/d/176wEroOVqI5V3xM-lf3SzMZ3ZPjOJMQeNEG3J9CnxJs/edit?usp=sharing

Please make your comments and write possible considerations, we will be attentive to give answers.

If we advance this phase we will work on the design and implementation as next steps.

jacobvanetten commented 1 month ago

To create functionality in ClimMob to manage these templates themselves through an interface seems a disproportionate effort, as we indicated in the SCRUM. We should maintain the templates as stable as possible, so this interface would almost never be used. Also, the proposed interface does not allow a link to ontologies. It is not clear why you have discarded the option to create a JSON through the CEDAR workbench, mentioned in the comments. This already exists and links to ontologies. You have not indicated why you propose an alternative option and you have not indicated how you will deal with ontologies. The proposal I made in the SCRUM was to focus on a workflow that creates a JSON in CEDAR and then converts the resulting JSON from CEDAR format to whatever would work in ClimMob.

El mar., 15 oct. 2024 1:01, Brandon Madriz @.***> escribió:

In the previous SCRUM (Thursday 10-10-2024) I talked about a proposal for the implementation of metadata using JSON files, the following is the documentation of the possible flow that this functionality could have.

Link: https://docs.google.com/document/d/176wEroOVqI5V3xM-lf3SzMZ3ZPjOJMQeNEG3J9CnxJs/edit?usp=sharing

Please make your comments and write possible considerations, we will be attentive to give answers.

If we advance this phase we will work on the design and implementation as next steps.

— Reply to this email directly, view it on GitHub https://github.com/mrbotcr/py3ClimMob/issues/312#issuecomment-2412481011, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGF5FPIKRZYYTNLGA2XN23Z3REMNAVCNFSM6AAAAABPKLBY2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJSGQ4DCMBRGE . You are receiving this because you were assigned.Message ID: @.***>

BrandonMrBot commented 1 month ago

The proposal is based more on how the flow would work in ClimMob, rather than focusing on the JSON, to clarify how the user will interact with the system and ensure we are on the same page.

The focus is not on the JSON itself, and the document speaks of it in general terms, precisely because its structure still needs to be fully defined, along with the consideration of ontologies, as you mentioned. However, we understand that whatever JSON format is decided upon, ClimMob should be able to interpret it.

The interface mentioned in the document refers to the process where you upload the JSON, assign it a name, indicate which types of projects should complete this form, and then store it. Without this interface, the JSON would need to be added manually to ClimMob, which reduces the platform's flexibility (This development is simple and quick to implement).

The software/platform where users create the JSON can be perfectly CEDAR.

Sorry for missing information @jacobvanetten

jacobvanetten commented 1 month ago

OK, clear now, you are proposing a simple interface dor uploading JSON metadata template files. But then a next step would be to take a look at CEDAR JSON output, I propose. We need to make clear what the entire workflow of the user would be before making implementation decisions.

El mar., 15 oct. 2024 20:31, Brandon Madriz @.***> escribió:

The proposal is based more on how the flow would work in ClimMob, rather than focusing on the JSON, to clarify how the user will interact with the system and ensure we are on the same page.

The focus is not on the JSON itself, and the document speaks of it in general terms, precisely because its structure still needs to be fully defined, along with the consideration of ontologies, as you mentioned. However, we understand that whatever JSON format is decided upon, ClimMob should be able to interpret it.

The interface mentioned in the document refers to the process where you upload the JSON, assign it a name, indicate which types of projects should complete this form, and then store it. Without this interface, the JSON would need to be added manually to ClimMob, which reduces the platform's flexibility (This development is simple and quick to implement).

The software/platform where users create the JSON can be perfectly CEDAR.

Sorry for missing information @jacobvanetten https://github.com/jacobvanetten

— Reply to this email directly, view it on GitHub https://github.com/mrbotcr/py3ClimMob/issues/312#issuecomment-2414731254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGF5FI5CGJYWM7LAUVONQDZ3VNOZAVCNFSM6AAAAABPKLBY2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUG4ZTCMRVGQ . You are receiving this because you were mentioned.Message ID: @.***>

BrandonMrBot commented 1 month ago

Perfect.

We agree with the next step, we can pass the Excel Trial Metadata to CEDAR format to analyze the output that is generated, without ontologies for the moment.

Do you know which ontology you want to use? I understand that CEDAR uses the ontologies from https://bioportal.bioontology.org/.

MarManrow commented 4 weeks ago

@jacobvanetten could you reply to Brandon´s comment, please?

jacobvanetten commented 4 weeks ago

This is a question for Marie. Crop Ontology is there, for example.

On Wed, Oct 23, 2024 at 9:22 PM Marilyn Manrow @.***> wrote:

@jacobvanetten https://github.com/jacobvanetten could you reply to Brandon´s comment, please?

— Reply to this email directly, view it on GitHub https://github.com/mrbotcr/py3ClimMob/issues/312#issuecomment-2433253221, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGF5FIMG4I6EM7QBCCZUELZ47ZQXAVCNFSM6AAAAABPKLBY2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZTGI2TGMRSGE . You are receiving this because you were mentioned.Message ID: @.***>

MarManrow commented 4 weeks ago

@marieALaporte could you give us feedback, please?

BrandonMrBot commented 4 weeks ago

For some reason when I try to access: https://bioportal.bioontology.org/ and I select Crop Ontology it shows the error:

Screenshot_20241023_160238