search

PollyNET / Pollynet_Processing_Chain

NRT lidar data processing program for multiwavelength polarization Raman lidar network (PollyNET)
https://polly.tropos.de/
GNU General Public License v3.0
19 stars 8 forks source link

Change plotting mode of time-height plots to 24 h #214

Open HolgerPollyNet opened 1 year ago

HolgerPollyNet commented 1 year ago

@ulysses78 , can you briefly discuss what you are currently working on, so that everybody is informed and link to the respective branch?

Thanks

ulysses78 commented 1 year ago

aim of this project

This project combines two objectives:

  1. The first goal of this project is to have 24h time-height plots for e.g. attbsc, voldpol, categorization, ... and so on. Until now those plots are splitted into 4x6h plots (in most cases). This is due to the measurement process. With the 24h-plots the diagrams can be interpreted better and more easily. Such 24h plots are also used by some other cooperation partners, e.g. cloudnet (fmi). Profile plots are not part of this project.
  2. The second goal is to speed up the python plotting procedure

picassos plotting procedure: status quo

Until now Picasso generates the plots not directly from the level1-nc-files, but from the processed results using matlab. Those results are temporarily stored in a .mat file. The content of this .mat file is then used for plotting. The plots are generated by calling python-scripts within matlab. The python plotting procedure at the moment uses the pcolormesh function. Advantage of this function is the nice handling of measurement gaps, but the big disadvantage is the slow processing speed. This is a critical major issue when processing and plotting huge amounts of polly data.

the new approach - 24h plotting

In the new approach one has to combine all the informations from one day. This is done by merging the level0 files first. The level0 files are only merged if the polly settings haven't changed during the day. Afterwards the merged file is proccessed by using picasso until the results are stored in the level1-files. The new plotting scheme uses the information stored in the level1-files. It uses the imshow python-function. The advanatage of this function is its fast processing speed (approx. 8x faster compared to pcolormesh). The disadavntage is the bad handling of measurement gaps. To avoid this disadvantage I used matrixfilling of the data blocks. In other words: for one day an empty matrix is created and is filled with the available measurement data information (e.g. attbsc, voldepol, ...).

ulysses78 commented 1 year ago

First test quicklooks

To view the actual running test quicklooks, Rico Hengst set up a local web server@rsd2 running at localhost:3333. To access this test instance of pollyAPP use portforwarding and create a ssh connection to Bildermacher2@rsd2 ssh -L 12349:localhost:3333 Bildermacher2@rsd2 If the connection is established, please use your local webbrowser http://localhost:12349/ and connect localhost at the named port. This test instance is a little bit slow, so please be patient.

ulysses78 commented 1 year ago

FYI: @ZPYin @rico-hengst @cristoferjimenez @AthenaAugousta @gast-ben @martin-rdz @Ronny-TROPOS

HolgerPollyNet commented 1 year ago

should this really be public?

elmarinou commented 1 year ago

Ioanna has a code to make tailored pocasso plots, where the user can choose time start, time end, height range, and colorbar max values for volume depol. and att. bp. Do you think we should consider this option as a new tailored plots topic? or can this code you are developing now also have an option like this? Have a nice weekend!

ZPYin commented 1 year ago

FYI: @ZPYin @rico-hengst @cristoferjimenez @AthenaAugousta @gast-ben @martin-rdz @Ronny-TROPOS

Hi Andi(@ulysses78)! Thanks for your efforts to make it work.

From my point of view, there are two different things in between.

PollyNET Processing Chain is an public repository to show our partners how to deal with PollyXT data. And pollyAPP is so far an internal application for showing pollynet-related content. This repository only focuses PollyNET Processing Chain staffs.

I will hide your posts in terms of pollyAPP. But free feel to communicate pollyAPP content through emails.

ulysses78 commented 1 year ago

I just mentioned this special version of the pollyapp, because there you can see the 24h plots for pollyxt_lacros and pollyxt_cpv.

ulysses78 commented 1 year ago

For security reasons I shifted the detailed description of this project to our internal doku-wiki pages (only accessible via vpn): https://rsdwiki.tropos.de/doku.php?id=data_processing:picasso:picasso24h

ulysses78 commented 1 year ago

Ioanna has a code to make tailored pocasso plots, where the user can choose time start, time end, height range, and colorbar max values for volume depol. and att. bp. Do you think we should consider this option as a new tailored plots topic? or can this code you are developing now also have an option like this? Have a nice weekend!

We also have a pypi-project called pylarda (https://github.com/lacros-tropos/larda). With this one can generate time-height plots for different devices (e.g.: pollyxt, hatpro, halo, mira35, rpg94, ...). But with pylarda it is not possible to plot quicklooks during the picasso-processing. The aim of this picasso24h-plot project is to generate 24h-quicklooks during picasso is processing the (new) measurement data and speed up the plotting process.

HolgerPollyNet commented 11 months ago

FYI: @ZPYin @rico-hengst @cristoferjimenez @AthenaAugousta @gast-ben @martin-rdz @Ronny-TROPOS

Hi Andi(@ulysses78)! Thanks for your efforts to make it work.

From my point of view, there are two different things in between.

* PollyNET Processing Chain

* pollyAPP

PollyNET Processing Chain is an public repository to show our partners how to deal with PollyXT data. And pollyAPP is so far an internal application for showing pollynet-related content. This repository only focuses PollyNET Processing Chain staffs.

I will hide your posts in terms of pollyAPP. But free feel to communicate pollyAPP content through emails.

Actually, the 24h capability is kind of a new fork from the PPC. So its fine to leave it here... Best