[Topic] Data ingress pipelines and preprocessing

[aranas]

Topic Data ingress and preprocessing pipelines

How is the topic relevant to the tric-dt themes?

How does it relate to the wider topic of digital twinning? Efficient data access, preprocessing, and real-time processing are foundational for the development and functioning of digital twins, ensuring that the models are fed with accurate and timely data.

Suggested speakers or contributors @ots22 could present some examples from work with BAS (Environment) Jose Lemus (Health)

Any resources you can recommend on this topic?

What format do you think would serve this topic best? part of seminar series

[aranas]

As discussed @alonsoJASL your work would be example for this topic. would you be up for giving a 15 minute presentation on this? what topics do you think would make most sense to cover here? Feel free to add more specific suggestions so others can see and react

[alonsoJASL]

I'm happy to give a 15 min talk on this.

The points you make are very relevant. Something that comes to mind based on the first two questions is the concept of coupling of data, which I take from telecommunications nomenclature. The idea is that between two processes there needs to be something that aligns them so that they can work together. There are sources of noise as a consequence of this coupling, which should also be addressed.

Items that I would consider important to cover:

• dealing with multiple inputs and multiplicity of formats
• writing software that is platform agnostic
• writing code that is easy to setup and has a low entry bar
• interoperability with the platforms down the pipeline
• maximising reproducibility: from code/programs to documentation/tutorials

[aranas]

@Ulvetanna also wondering if aspects of PolarRoute could be good examples here?

[aranas]

Thank you @alonsoJASL also for providing some more detail. I'm thinking that @ShenXiaoxue work on Ontologies will likely also link with interoperability

[ots22]

Happy to present some of the work with BAS (need to think about exactly what I'd say!)

[aranas]

@mhauru tagging you here as you are currently working on data ingress pipelines for DTBase. Would you be interested in participating in this session?

[mhauru]

Yep, sounds good, thanks!

[ShenXiaoxue]

If we have any chance to invite Chris Baker to have a coversation on KG and ontologies, that would be great!

[aranas]

opening up a question that came up in a communication with @mhauru and @ShenXiaoxue. Designing a good relational database that is general enough to be interoperable with different data types is a challenge. Knowledge graphs also embed relational information and are usually based on some relational database. What are the steps from database to knowledge graph and what are best practices that should drive the design of the DB from the start? I think these questions might form a great basis for a joint session. This also matches with @alonsoJASL 's point about "multiple inputs and multiplicity of formats"

[JimCircadian]

@ots22 @aranas also happy to discuss from the BAS side, fling me a message if you'd like any input!