search

PollyNET / Pollynet_Processing_Chain

NRT lidar data processing program for multiwavelength polarization Raman lidar network (PollyNET)
https://polly.tropos.de/
GNU General Public License v3.0
20 stars 8 forks source link

renew gdas takes too long #83

Closed HolgerPollyNet closed 3 years ago

HolgerPollyNet commented 3 years ago

Hi@all, meanwhile the renew gdas script running at mindnight takes 15 hours to finish ;-) 96 task ar ei the queue. We need tu change that, because no current data is processed during this time. E.g. today data from 12 to 15 November (each day 4 files) for 7 pollys is repcrocessed. This is 7 times 4 times 4=96 6-hour files. One files takes for compelte processing about 10 minutes, are 960 minutes = 16 hours...

my proposal: a) do not reprocess the last 4 days but only the the last day in the queue. e.g. in the example aove from 12 November. b) speed up plotting ;-) c) only reprocess data affected, e.g. optical profiles and quasi parameter and RH d) any other idea....

ZPYin commented 3 years ago

I need to point out that usually there is only 1-2 days data with corresponding gdas profiles taking into account of the delayed time of gdas data. Therefore, the reprocessed data is not as big as you thought. So, (a) may not be too helpful although it's the easiest to be implemented.

(b) should be the final choice, and Martin and I have discussed it in issue #75.

(c) is possible with some minor changes, but i doubt about the speed improvement due to more than half of the plots need to be reprocessed. Anyway, we can give it a try.

PS,

I wonder whether it's possible to grant me the access to rsd2. might be good to check this issue in detail. as far as i can see the plotting routines seems five times slower than in old server ( see #75 ), which i think is the main reason for jamming the whole processing chain.

ZPYin commented 3 years ago

One possible reason for this, as I checked in the new server, is the conflicted cron job by routine processing and gdas reprocessing at every 01:00. I have removed the routine processing at 01:00 to leave the computation power to gdas reprocessing, in order to increase the processing efficiency. Let's see whether it helps.

ZPYin commented 3 years ago

One possible reason for this, as I checked in the new server, is the conflicted cron job by routine processing and gdas reprocessing at every 01:00. I have removed the routine processing at 01:00 to leave the computation power to gdas reprocessing, in order to increase the processing efficiency. Let's see whether it helps.

It does not help at all... But I have shrink the search date from past 4 days to past 4 days to the day before yesterday. It's a temporary solution. I will try to resolve it in the new version of Picasso.

HolgerPollyNet commented 3 years ago

I do currently not have any problem with this issue anymore.

ZPYin commented 3 years ago

OK, then I will close this issue. But feel free to re-open it when it happens again.