odtp-org / odtp

ODTP: A tool designed to manage, run, and design digital twins.
https://odtp-org.github.io/odtp-manuals/
GNU Affero General Public License v3.0
12 stars 1 forks source link

[FEATURE] Data storage #1

Open zuocsfm opened 1 year ago

zuocsfm commented 1 year ago

04.12.2023

Requirement

Data storage

Description:

A spatial database with a GUI that can support researchers to publish, update, describe, and exchange the datasets they have collected. It should allow the data owners to manage permissions for potential users.

Importance Level

High

Origin

It requires from multiple stakeholders, like Prof. Dr. Kay W. Axhausen, Institute of Construction & Infrastructure Management, ETH Zürich, Geographic Information Systems, University of Zürich, etc.

User Impact

This feature will accelerate the research in data and ideas exchange. It can foster potential collaborations and expertise exchange between researcher groups. Database maintenance and administration will be a potential drawback. However, it does not require much workload.

Mockups or Diagrams

Not applicable.

Affected Components (examples: components, modules, … )

It can be linked to other components and modules by APIs, or an standalone function in the starting stage.

Technical Requirements (if possible, otherwise completed by SDSC)

Detailed technical specifications or requirements needed to implement the feature. This could include algorithms, data structures, APIs, or third-party services.

Related Documents/Links:

Dependencies:

The spatial database can interact with existing ODTP components, like visualization component and future data analysis component.

caviri commented 1 year ago

Hello @zuocsfm, thanks for writing down the request. We'll come back to you with some improvements and questions soon.

Before that, please try to correct the ticket to the right formatting in markdown and use a tittle. The template in Github provides the different parts with already preconfigured markdown formatting.

Thanks, Carlos

zuocsfm commented 1 year ago

04.12.2023

Requirement

Data storage

Description:

A spatial database with a GUI that can support researchers to publish, update, describe, and exchange the datasets they have collected. It should allow the data owners to manage permissions for potential users.

Importance Level

High

Origin

It requires from multiple stakeholders, like Prof. Dr. Kay W. Axhausen, Institute of Construction & Infrastructure Management, ETH Zürich, Geographic Information Systems, University of Zürich, etc.

User Impact

This feature will accelerate the research in data and ideas exchange. It can foster potential collaborations and expertise exchange between researcher groups. Database maintenance and administration will be a potential drawback. However, it does not require much workload.

Mockups or Diagrams

Not applicable.

Affected Components (examples: components, modules, … )

It can be linked to other components and modules by APIs, or an standalone function in the starting stage.

Technical Requirements (if possible, otherwise completed by SDSC)

Detailed technical specifications or requirements needed to implement the feature. This could include algorithms, data structures, APIs, or third-party services.

Related Documents/Links:

Dependencies:

The spatial database can interact with existing ODTP components, like visualization component and future data analysis component.

caviri commented 1 year ago

Hello @zuocsfm, thank you. You can edit your preview messages and the title by clicking on the three dots next to your message. I edited the message and hide you previous message as duplicated. I also added the title for you as Data storage. Feel free to edit it.

ksanao commented 12 months ago

Hello @zuocsfm,

We have reviewed the scope and feasibility of this feature request.

While this feature aligns well with our project's objectives, its development presents significant scope and resource considerations. Given the complexity and the amount of work involved in creating a dedicated database, such a task typically requires a separate, dedicated project and team. We estimate several months of work on requirement refinement and about 6 months of work for a team comprised of a database developer, a GIS specialist, an integration engineer, and a data analyst. Considering our one-year timeline for our current project, a more efficient and pragmatic approach might be to leverage existing spatial data resources and databases for integration.

This feature aligns more closely with the expertise and work packages of CSFM team, especially since it does not directly relate to the core components of the ODTP but could be integrated via APIs or as an initial standalone feature. Would CSFM be open to leading the development of this feature? We at SDSC are fully prepared to provide comprehensive support in guiding its integration within the ODTP framework. By aligning our efforts, we can maximize our strengths and ensure a streamlined development and integration process.

Thanks for your collaboration on this; we're keen to hear your thoughts.

caviri commented 12 months ago

I agree with Oksana's evaluation, the development of a spatial database represents a project by itself. Which, as Oksana said, can be integrated with the ODTP via API calls in dataloaders components.

Based on your references it seems that you may be interested in having a database that provides this information to visualization apps, is this correct? If so, one solution could be to generate a digital twin with one component that contains the spatial database and the visualization app. Here you have a diagram of how it would looks like. In this case ODTP can retrieve the data from the original source(s) and update the spatial database that works as the app backend.

image (12)

Also, something to consider about storing is the internal s3 storage solution that odtp have to store outputs (results/snapshots). These can be shared across multiples digital twins of the same instance (If the permissions allow it).

Finally, I think that this request will improve with a more detailed description. For instance, providing example datasets that are expected to be hosted, how the users would like to access them (i.e. via downloading individual files, accessing to chunks of data based in coordinates, or streaming data), and the acceptance criteria.

I hope this helps, we can discuss more in the our meeting.