Plotly/HoloViz/FiftyOne widget suggestions + bug list

[dissagaliyeva]

Hi everyone! I think I'm a little late for the party :)

I'm both super excited and terrified to be hopefully joining this project!

After playing around with the functionality, I saw that from raw to cleaned files, lat/long get lost on the way. I was going to suggest a workaround to use them for coordinate representations but I see all of the information is covered already in #26

Expanding on #18 and #27, there are two solutions that would be beneficial for echoshader's advancement. The first one would benefit from dynamic closeable tabs. The two tab versions are from VoloViz & select option from plotly: 1) Dynamic tabs 2) Closable tabs 3) Plotly selection

It’s essentially built on top of @ldr426 idea explained in #27.

The idea is the following: In the first tab have a map of the world (or of a specific location) along with sonar data from the surface view as explained in issue #17. It would have several functionalities: select region, time, and ping (or anything that is found useful). Here we would have the functionality of selecting as shown here using Plotly selection

The second tab would show the characteristics and necessary information as explained in #18. The idea is to have dynamic tabs so that it would have the flexibility to select as many or as few regions as necessary to have a direct comparison. When the user is done exploring the specifics, they can simply close them. Also, if necessary save them as an image.

It would essentially look like this:

=====

The second solution will require a lot of work but will give a better result in the end (in my opinion). There's such a package called fifty-one that primarily deals with computer vision tasks. What is cool about this app is that it can solve A LOT OF tasks and has a great interface that can be re-programmed to our needs. Moreover, it doesn't run on another page but has an interactive widget. There's a pretty big degree of freedom that doesn't limit users, and as suggested in issue #27, it's the main goal.

The bad news is that it works only with images, so it's not going to be directly applicable. So, we could essentially create an app in the same fashion but tailored for our needs, of course. Here are a couple of links to check out the app:

1. Notebook to understand how it works
2. Functionalities

=====

Some bugs I found along the way:

1. Broken link Where: link

Under:

echopype contains tools for converting data from various manufacturer-specific formats into a standardized [netCDF]() files

The broken link: link

=====

2. Missing NumPy dependency Where: link

When downloading with git clone and setting up the environment, the code breaks when running the following block:

MVBS_ds["Sv"].hvplot.image(

    x='ping_time', y='range', 

    color='Sv', rasterize=True, 

    cmap='jet', clim=(-80, -50),

    xlabel='Time (UTC)',

    ylabel='Depth (m)'

).options(width=500, invert_yaxis=True)

Error message: ImportError: Datashading is not available & ImportError: Numba needs NumPy 1.20 or less

=====

3. Wrong setup directory Where: link

The README.md file contains the line conda env create -f environment.yml but the environment.yml file is in the binder folder. It should be replaces with conda env create -f binder/environment.yml

=====

There is one functionality in fifty-one that lets us use geospatial data with this package, so I'll experiment with it and give feedback afterward along with the notebook so that you can easily reproduce the results :) There are going to be several of them that will also test the first suggestion!

Please let me know if any of these ideas are applicable!

P.S., I’m super excited to meet you all! And sorry for a huge message😅

[ldr426]

It’s essentially built on top of @ldr426 idea explained in https://github.com/OSOceanAcoustics/echoshader/issues/27.

:smile: Thank you for rebuilding my idea, Dinara. But this idea has been abandoned. We don't need to create a certain template for users in echoshader. Users can select panel components to combine what they want. Creating some interactive APIs would be better than a template.

Error message: ImportError: Datashading is not available & ImportError: Numba needs NumPy 1.20 or less

By the way, the error could come from your environment. It runs without errors in my jupyter. Datashading is a package for visualizing big data in the holoviz ecosystem. You could pip install datashader and NumPy == 1.20.

Hope you can have some good results using the package fifty-one :)

[dissagaliyeva]

Hi @ldr426, thanks for the reply! that's interesting, I had no idea the idea for the panel had been abandoned already.

By the way, the error could come from your environment. It runs without errors in my jupyter. Datashading is a package for visualizing big data in the holoviz ecosystem. You could pip install datashader and NumPy == 1.20.

Yup, I was simply referencing that numpy was missing in the requirements.txt file :)

Hope you can have some good results using the package fifty-one :)

Thank you! And you as well, hope everything is going well with that API you're working on!

[lsetiawan]

Hi @dissagaliyeva, could you tell me a little bit about your working environment. What operating system are you using? Also for your conda environment, what python version are you on? Thank you.

[dissagaliyeva]

Hi @lsetiawan! Sure, I'm on Windows 10 and Python version 3.9.7

[emiliom]

echopype contains tools for converting data from various manufacturer-specific formats into a standardized netCDF files

The broken link: link

Thanks for reporting this broken link. We'll fix it soon. In the meantime, you can find a lot of information about netCDF online, but the netcdf web site is https://www.unidata.ucar.edu/software/netcdf/. In practice, you will be using a higher-level Python library to interact with netcdf files, especially xarray. This xarray User Guide page is a nice and compact introduction to the data structure.

[dissagaliyeva]

@emiliom thank you for the info!

[leewujung]

Hey @dissagaliyeva, welcome!

that's interesting, I had no idea the idea for the panel had been abandoned already.

I wanted to chime in here on the overall idea. The panel idea actually has not been abandoned, it just takes a form that’s perhaps different from the regular web portals.

What we aim to create in echoshader is a suite of “widgets” that can be mixed and matched by users to configure their own visualization dashboards using the Panel library. This is because regular researchers may not necessarily have the human resources to help deploy and maintain their own front and backend services, so we want to enable some more flexible and convenient data viz capability that can be used in a Jupyter notebook for example.

This does mean that we need to provide the data linkage between these widgets since the interactions often would require slicing based on specific coordinates. For these “under the hood” data handling operations we need to make decisions on whether it goes in echoshader here or echopype. So some planning and more discussions are needed on this front.

[dissagaliyeva]

Hi @leewujung, thank you a lot for your clarification, that makes more sense!

Also, I want to update on the second idea, after spending almost a week on that. As anticipated, the default parameters didn't work as intended & making a workaround also raised too many issues. This experience showed that using HoloViz's panel is indeed a better choice due to its simplicity and rich functionality. Therefore, I'll disregard the second option and primarily focus on the first one :)

I wanted to chime in here on the overall idea. The panel idea actually has not been abandoned, it just takes a form that’s perhaps different from the regular web portals. Awesome! Widget it is, I'll be pushing the results shortly!

[dissagaliyeva]

Hi @leewujung!

Sorry, it took me some time to understand how to make an inline working prototype for the idea. Essentially, I was following the prototype I've had in mind:

The prototype ended up being similar to the idea, but there were two changes along the way: 1) Dynamic and closable tabs can't be combined together, so the workaround was to create two separate non-closable tabs that hold the default route map and results. The results tab has sub-tabs that can be closed as needed. Essentially, it looks like this for the moment:

2) I can't decide between a dropdown and a RadioBoxGroup, so both implementations are present in the notebook. However, only one of them is interactive and affects the frequency map created. I've labeled them accordingly :)

===

A couple of things to know about the prototype: 1) At first, I wanted to use a beautiful approach of DatetimePicker but it doesn't really work well with hours/minutes/seconds. It allows selecting them but the information gets lost along the way. Therefore, for now, I went with the simpler (and uglier) option to use DatePicker and FloatSlider.

Also, I've disabled any other days from being selected that is not present in the files that were read. And I used the time range option that only accounts for the hours used in the files. I will need to think of a workaround that will not let choosing time before and after the minimum and maximum time ranges if there are multiple days.

2) A frequency map is dynamic and reacts to the selected frequency as intended. Frequencies are also created depending on the *.nc files parameters, which means if a file has 3000 and 5000, it will only have those two values.

3) There are some additional functions created. The descriptions are below, for convenience they're also present in the notebook.

• combine_data

This function takes in all the *.nc files and combines them. The problem with ep.combine_data function is that it doesn't catch the errors when given inappropriate data formats. That is especially evident if there are files like Summer2017-D20170625-T190753_Sv.nc (that end with _Sv) and Summer2017-D20170625-T190753.nc.

For my purpose, I've created a separate function that ignores files with no latitude/longitude and combines all of the rest. For now, there are no explicit tests created, it's just a working prototype for the current data in hand. I'll be using lat/lon to create a GeoPandas & further create visualizations for maps and frequencies.

• get_geo & compute_mvbs

The get_geo function creates a DataFrame that holds lat/lon as used in echopype_tour. The other takes advantage of the ep.calibrate.compute_Sv functionality but created as a function to overcome redundancy.

• show_freq

This function shows the maps depending on the selected frequencies.

• get_map

This function creates a map that holds all of the traces using Plotly. Ideally, this function will have the capabilities to select using both box-select and lasso-select. For now, the function simply shows the map interactively.

====

Problems

There's one more problem that I'm facing. I can't solve the box- and lasso-select for now. Since there's Plotly's map to show the route, both selection options are already there but they're not reactive yet. This one will need more time.

[lsetiawan]

We have selected a contributor and now in a working phase. Closing this issue. Thank you all for your thoughts and ideas in the brainstorm phase.