Assignment 1 [Alessandro Longhi]

Aleartulon commented 1 year ago

Introduction

Describe your research in 2-3 sentences to someone that is not from your field (please avoid abbreviations)

My research entails the following aspects:

Research Aspect	Answer
Use/collect personal data (health data, interviews, surveys)	No
Use/collect experimental data (lab experiments, measurements with instruments)	/No
Collaborate with industry	Yes
Write/develop software as the main output of the project	Yes
Use code (as in programming) for data analysis	Yes
Work with large data (images, simulation models)	Yes
Other:	N/A

Reflections on the importance of RDM videos

<- Data management is important especially because it is something that is generally not taught, and young researcher could think it is something not useful, until something bad happens. Nothing bad has happened to me, but in my lab by mistake a laptop was unplugged by the charger and all data were lost.->

What would you like to learn during this course?

Checklist assignments

[x] Assignment 1: creating a GitHub issue (before 12 September, 10:00)
[x] Respond to the GitHub issue #65
[x] Assignment 2: Data Flow Map 1 (share a link in this issue before 20 September 13:00). Link: https://surfdrive.surf.nl/files/index.php/s/rVvXKdWvIIF8sG5
[x] Provide feedback to at least one Assignment 2 from another participant
[x] Respond to the GitHub discussion #66
[x] Respond to the GitHub discussion #67
[x] Assignment 3: Data Flow Map 2 (share a link in this issue before 4 October 13:00). Link:
[x] Provide feedback to at least one Assignment 3 from another participant
[x] Assignment 4: Data Management Plan (before 11 October, 13:00)
[x] Respond to the GitHub discussion #68 if you have any questions (optional)
[x] Assignment 5: Data Flow Map 3: submit your slide (before 17 October, 17:00)

EstherPlomp commented 1 year ago

Hi @aleartulon! Thanks for submitting your first assignment! Can you try to edit your 'issue' using the three dots in the right corner of it? Then if you remove all the < !-- in the text, it will also show up in the preview/submitted version. Now it is hidden.

I hope that makes sense - let me know if you need any help with that.

EstherPlomp commented 1 year ago

Copied from #98:

Hi, this is the link to my template

https://surfdrive.surf.nl/files/index.php/s/rVvXKdWvIIF8sG5

giacomolastrucci commented 1 year ago

Hi Alessandro,

looks good! The only question that I have, given also the common framework in which we will be working, is: Do you already have such training data from numerical simulations or you have to actually run the simulations to get data? Can you consequently have multiple datasets (e.g., different operative ranges, etc.) or you just have to store one dataset?

Aleartulon commented 1 year ago

Hi Alessandro,

looks good! The only question that I have, given also the common framework in which we will be working, is: Do you already have such training data from numerical simulations or you have to actually run the simulations to get data? Can you consequently have multiple datasets (e.g., different operative ranges, etc.) or you just have to store one dataset?

Hi Giacomo, thank you!

In principle other partners inside my PhD projects are going to provide the data, but I also have the code to run the simulations myself. I think at the beginning I will use data from other people but at some point it will be easier for me to decide which data obtain in order to improve the ML predictions. So yes, it is very likely I will have to store multiple data-set, or at least I will have to access different data set from clusters of other research groups.

EstherPlomp commented 1 year ago

Thanks for sharing your assignment @Aleartulon, well done!

I think a couple of things could be further clarified:

With webdrive, do you mean the network drives of TU Delft? So the Group/Bulk and Project drives?
I don't see any file format and sizes listed: particularly the file formats will be important for assignment 3!

And, as you're working with commercial data, GitLab could be a more secure storage solution compared to GitHub. For the TU Delft GitLab instance everything is managed by TU Delft instead of GitHub/Microsoft that could use the code for their own purposes.

Hope that helps!

Aleartulon commented 1 year ago

Hello, please find here my 3 assignment! Let me know if you have comments/suggestions.

https://surfdrive.surf.nl/files/index.php/s/gLVQzDnsxmvy954

Aleartulon commented 1 year ago

Thanks for sharing your assignment @Aleartulon, well done!

I think a couple of things could be further clarified:

With webdrive, do you mean the network drives of TU Delft? So the Group/Bulk and Project drives?

I don't see any file format and sizes listed: particularly the file formats will be important for assignment 3!

And, as you're working with commercial data, GitLab could be a more secure storage solution compared to GitHub. For the TU Delft GitLab instance everything is managed by TU Delft instead of GitHub/Microsoft that could use the code for their own purposes.

Hope that helps!

Hi, thank you for the comments.

Yes, by web drive I mean the Project drives.
I added the file format in assignment 3 :). For the sizes is a bit difficult to know as I am still at the beginning and it is a bit uncertain.
Thanks, I will look into GitLab then!

EstherPlomp commented 1 year ago

Thanks for sharing Assignment 3 @Aleartulon! And for responding to the feedback! No worries if you're not sure about the sizes yet, estimations are more than sufficient! And well done on the rest of the assignment - it looks clear!

Some pointers from my side:

Metadata

Are there any dependencies and libraries that you use for the Python code? These can also play a role in reproducibility.

File Formats

HDF5 is also an open file format, and Python is indeed open as well!

Access

Google drive can be used for 'regular' research data but should be avoided for data of potential commercial use indeed

Data Publication

For data publication the active storage solutions like DropBox/Google Drive (or the TU Delft solutions like the project drive, SURF/OneDrive) are not sufficient - these are meant for during the project and not for data publication (see the section on Realising FAIR - Data publication). You'll need to make use of a data repository to share the data where possible - this may indeed not be the case for some of the work you do. I would think that the training data could be made available but then indeed not the software/code.
GitHub is an excellent solution to manage and store code, as well as to share it with external collaborators. However, it is not a long term solution as GitHub does not have a long term preservation policy and it also doesn't assign DOIs. For this there exists an integration with Zenodo, where you save a snapshot of the code to the Zenodo data repository. You can find out more about that here: https://docs.github.com/en/repositories/archiving-a-github-repository/referencing-and-citing-content

giacomolastrucci commented 12 months ago

Hi Alessandro, well done, a very good overview of your project data in Assignment 3. I think you have already got the most important feedback from Ester. I can just suggest you another tool (besides Tensorboard) to keep track of your training statistics, models and metadata. It is called Weights&Biases: it is very convenient, easy to implement, cloud-based and very comprehensive. Check it out!

EstherPlomp commented 11 months ago

That looks interesting @giacomolastrucci! thanks for sharing!

EstherPlomp / TNW-RDM-101