search

MotionbyLearning / stm-dashboard

Create a dashboard to visually inspect STM datasets
Apache License 2.0
0 stars 0 forks source link

improving example notebook #2

Closed rogerkuou closed 10 months ago

rogerkuou commented 10 months ago

Some ideas to improve the example

rogerkuou commented 10 months ago

Hi @fnattino , I made this PR purely to visualize some suggestions on the example notebook. Feel free to merge it or not.

I think the existing version looks good! Just minor suggestions on:

  1. Some suggestions in envrionment.yaml. I changed the name since I already have a mobyle env for dev. Since I thought this is something dedicated for visualization, so I renamed it. I need to install jupyter_bokeh using mamba to visualize the plot
  2. I changed the point plotting cell a bit to visualize the pnt_linear (linear regression of deformation rate in time) column with jet_r (reversed jet) colormap and a (-0.005, 0.005) (-5mm/year to +5mm/year) colormap. This is a common visualization giving general impression on deformation patterns in the region. read/yellow areas draw attention.
  3. Unfortunately I failed to generate Zarr file from the original 13G csv. Got a memory issue with my local laptop. I will look into this later. Do you think we should make the zarr file available on Spider, in case people would like to try this?
fnattino commented 10 months ago

Thanks for the feedback @rogerkuou! I will merge the PR and probably overwrite my notebook with your one.

About your point 3. : I have run the last demo on Spider, where the Zarr file is available. Considering that the demo on Friday will again be on Spider there should be no problem, right? Will use the Zarr file as a prerequisite and skip the conversion from the CSV file (which takes time but should be rather straightforward - provided you have enough memory).

rogerkuou commented 10 months ago

@fnattino About point 3: Indeed let's still try this on Spider. I just did not find the zarr file in the public path: /project/caroline/Public/demo_mobyle/. Maybe it sits in your home dir? Let's also move the zarr file to the data dir maybe?

fnattino commented 10 months ago

You are right, the Zarr file was in my home directory.

Hi have tried to save it in touch /project/caroline/Public/demo_mobyle/data/depsi_products (next to the corresponding CSV file), however I have noticed I don't have the write access to that subfolder:

touch /project/caroline/Public/demo_mobyle/data/depsi_products/tmp.txt
# touch: cannot touch '/project/caroline/Public/demo_mobyle/data/depsi_products/tmp.txt': Permission denied

And in fact:

ls -ld /project/caroline/Public/demo_mobyle/data/depsi_products

Shows drwxr-xr-x+ - so users in the same group don't have write access. Is this something intentional? Or can you otherwise run chmod g+w /project/caroline/Public/demo_mobyle/data/depsi_products to grant write access to the group?

Also, a side thought on the usage of the Spider filesystem: are you aware that all the files in /project/XXX/Public are also accessible by anyone on the web (see docs)?

rogerkuou commented 10 months ago

@fnattino you are right. I added group writing access to that directory. Can you check again?

And I was not aware that Public directory is accessible online. Will check with Freek.