Clean up notebooks

[raspstephan]

Hi Hauke,

I think we should clean up the notebook structure a little. Like merge the two folders we have right now.

I briefly went through all the notebooks to see what is done in them:

In analysis:

• agreement_practice: Test IoU score
• basic_analysis: Your analysis notebook with user statistics (time taken, distributions for regions, some others)
• IoU: Explanation and simple examples for IoU score.
• IoU_multi_user: Cross-user IoU
• organize-raw-dataset: Basic reading and preparation of raw Zooniverse dataset.
• zooniverse_data_sqlite_import: data loading and some analysis on the practice dataset (how many labels for each image)

In notebooks_SR:

• Several notebooks with my previous machine learning attempts: NWPU-BB, keras_test, pascal ..., segmentation, retinanet, cloud-label-demonstration (This one contains the gridded approach and data preparation)
• Zooniverse-EDA: Using Zooniverse data, basic manipulation plus plotting

I would suggest to put the current notebooks into one folder called notebooks (for now, we can later on subdivide it). All not current notebooks (like the early ML ones), I will put in a subfolder called old_notebooks.

Then we should merge some notebooks. All IoU stuff should go into one notebook. All basic analysis (i.e. everything that does not require images or ERA data) should go into one notebook. All data wrangling should go into the organize... notebook.

Utility functions that are used more than once should go into pyclouds/helpers.py. I hope that our work here will be used by others at some point (e.g. at a hackathon with the data), so I think it's great to write nice code (as you are already doing, I need to force myself to write proper docstrings).

Does all that sound good?

[observingClouds]

Yes, that sounds all good! I try to keep to that structure

[raspstephan]

I started the restructuring. Will continue tomorrow. I went through all your analysis and tried to write handy functions to ease the data wrangling. I also tried to do as much as I could with pure Pandas.

Tomorrow, I will start with some analysis on user agreement. Trying to come up with some easy-to-understand agreement metrics.