flickering and slowdown/memory issues

[falkamelung]

Hi @stackTom

I think I have figured tout the disk filling up issue. There were old docker images, sudo docker system prune -a removes them.

I installed it on two instances (tileserverDevelopment branch) to ckeckout the impact of memory:

149/165.168.186: 3GB RAM 1 CPU, 
149.165.153.50: 15GB 4 CPUs 

and ingested a big dataset (15GB):

149.165.168.186/start/39.5/118.4/11.0?flyToDatasetCenter=true&startDataset=S1_IW123_069_0126_0130_20141022_20231116_N39495_N39710_E118090_E118410_filtDel4DS
149.165.153.50/start/39.5/118.4/11.0?flyToDatasetCenter=true&startDataset=S1_IW123_069_0126_0130_20141022_20231116_N39495_N39710_E118090_E118410_filtDel4DS

and a smaller one: 800 MB

149.165.168.186/start/-8.5/123.6/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_090_0622_0624_20170116_20240807_S08552_S08486_E123555_E123591_filtDel4DS
149.165.153.50/start/-8.5/123.6/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_090_0622_0624_20170116_20240807_S08552_S08486_E123555_E123591_filtDel4DS

If I don’t do anything fancy it works fine. But for now I observed these problems>

1. If you change pixel size and then go to the layer button it starts flickering.

2. If you select a shorter time period and then move around and then select a point I see

For me this is easily reproducible. Go to :

149.165.168.186/start/-8.5/123.6/11.0?flyToDatasetCenter=true&startDataset=S1_IW3_090_0622_0624_20170116_20240807_S08552_S08486_E123555_E123591_filtDel4DS

Sometimes it get scrambles when I select the start date and other times you have to move around a bit.

3. When I select a limited time period and just move around without selecting a pixel and/or changing the color scale the "Recoloring in progress box" shows up all time. Maybe something is not efficient? The same happens when a reference point is selected manually. If we could speed this up that would be fantastic. (If nothing can be done about this I wonder add a feature to change the reference point. The workflow is, we first ingest, look at the data and then select the reference point accordingly. Once we know which is a good reference point we could run an additional json_2_mbtiles command to change the reference point).

4. I am trying to understand teh memory requirements. When I open the big data set on the small memory instance it sometimes goes very fast (even after emptying cache), and sometimes it take a few minutes. Why could that be?

   149.165.168.186/start/39.5/118.4/11.0?flyToDatasetCenter=true&startDataset=S1_IW123_069_0126_0130_20141022_20231116_N39495_N39710_E118090_E118410_filtDel4DS

   Please explain a bit what operations are done on the remote instance and which in your browser to understand memory. Given the memory available on insarmaps, should we allow datasets with limited size only?

5. Both, changing pixel size and selecting a reference point id currently a mess, but will probably work after fixing (1).

[stackTom]

Regarding 3 - we only recolor those points currently in view and hide the rest. For performance reasons. Otherwise the recoloring is impossibly slow. Whenever the map is changed, we must recolor the new points that are now in view.

Regarding the scrambling, it appears postgresql has changed how some commands work. We were on an old version, 9.2 in our insarmaps server. Now in docker, we are on 16. I will have to fix the sql commands I use. Or just roll back the Postgres on the docker to version 9.2.

[falkamelung]

Regarding 3, yes, we talked about that before. I just want to be sure we are on the same page regarding this. Even if I just more around without zooming it happens. See video. In this case there is no need to recolor anything, is there? https://github.com/user-attachments/assets/64380f24-9368-4db5-aa3a-70bf065c613a

It occasionally also happens just after displaying a dataset without selecting a time period (I think). In that case there is no reason to recolor? This is more difficult to reproduce, but I can try if needed.

And, finally, if nothing can be done abut this, would a big memory machine mitigate this? Jetstream also has some GPUs which we could try.

And regarding the scrambling, if you find the time to fix this, my preference would be stay with version 16

[disilvestro]

Hi everybody, I would like to address some more issues.

• [ ] Sometimes, once the Select Reference Point option is selected, the option remains active and there is no way to turn it off.
• [ ] When you show 2 timeseries together by choosing the option Second graph, once you deselect it, the second timeseries still runs in the background. What happens is that when you select a new point, the timeseries in the background(not visible) changes but not the visible one.
• [ ] Some swaths are not visible until you zoom in

[falkamelung]

Hi @stackTom and I tried to update the insarmaps.miami.edu. The repo is in docker/insarmaps.new, correct? I was not sure about the locations of the postgres_dir and mbtiles_dir. Which ones are we using? See below the /data listing.

Once we are finalized I probably will move or restart as /data/postgres_dir /data/mbtiles_dir for consistency with the other instances.

[insaradmin@insarweb insarmaps]$ ll /data
total 48
drwxr-xr-x  3 insaradmin root        4096 Apr 23 14:22 converting
drwx------  2 insaradmin root       16384 Jan 24  2017 lost+found
-rw-r--r--  1 insaradmin postgres   16374 Dec 21  2017 OUT.txt
drwx------ 19        101        104  4096 Sep 21 19:57 postgresDBData
drwxrwxr-x  2 root               33  4096 Sep 21 18:07 tileserver
drwxrwxr-x  3 insaradmin insaradmin  4096 May  9  2020 TILESERVERBACKUP

[stackTom]

I rolled back to postgresql 9.2 and the scrambling is fixed. I will have to investigate what changed between 9.2 and 16 that causes the query from the database to return different results. I'll look into the other issues as time permits. Best to put every bug/feature request as a new github issue, or it will get confusing to answer everything in one issue

[stackTom]

Regarding 3, yes, we talked about that before. I just want to be sure we are on the same page regarding this. Even if I just more around without zooming it happens. See video. In this case there is no need to recolor anything, is there? https://github.com/user-attachments/assets/64380f24-9368-4db5-aa3a-70bf065c613a

It occasionally also happens just after displaying a dataset without selecting a time period (I think). In that case there is no reason to recolor? This is more difficult to reproduce, but I can try if needed.

And, finally, if nothing can be done abut this, would a big memory machine mitigate this? Jetstream also has some GPUs which we could try.

And regarding the scrambling, if you find the time to fix this, my preference would be stay with version 16

I see what you mean. Yes, there is room to make this more efficient. More memory means postgresql can keep more stuff in RAM and have to read from the hard drive less, so it always helps. However, I recommend to upload to the insarmaps server. Isn't it more powerful than the exos server?

[falkamelung]

Yes, I will separate the other items into different issues. I started as an email and thought about gihub...

Making what we currently have more efficient would be the top priority. The user experience with the European site is excellent, but they don't have the recoloring nor select-reference-point features.

insarmaps.miami.edu had 8 GB Ram while the others have 3 and 15 GB. I can get an instance with more RAM if needed. I want to go back but it would be good to first understand the memory issue.

What you say about memory is what I thought. So as long as the full dataset is loaded into memory we might be able to do get an instantaneous response for recoloring/select-reference-point? That is what I am hoping for. One question is what is the maximum file size for a 15GB instance? We could adjust the processing accordingly.

• Now, after you rolled back postgres, how do I update? Is there a git docker pull or do I need to stop the current container and run init_docker' and 'run_docker.sh again? So we can't keep the current postgres_dir and mbtiles_dir?

[stackTom]

The slow parts of the site are anything that involves on the fly recoloring. This includes the date select feature as well as the reference point feature. I am not sure how to make these e any faster aside from just having a more powerful server, I'm afraid. I tried to optimize it the best I could.

It's hard to say what the max file size is. It depends how many users are using the site simultaneously, etc, and what the database is doing (whether ingests are occuring, etc).

In regards to updating - any time I update, all you need to do is do a git pull. then stop the current container, and re-run init_docker.sh and run_docker.sh. I had to empty the postgres_dir and mbtiles_dir, but for future updates, we can re use this data just like when the server did not run docker.

Please ingest some data to double check the scrambling is gone. I did not see it again after going back to postgres 9.2.

[falkamelung]

I updated on insarmaps3 (ending with 50) and re-ingested but still see the flickering. Just adjust once the pixel size. Also insarmaps2 (ending in 186) which I suppose you updated still has it, unfortunately...

http://149.165.153.50/start/39.6027/118.2513/11.0000?flyToDatasetCenter=false&startDataset=S1_IW123_069_0126_0130_20141022_20231116_N39495_N39710_E118090_E118410_filtDel4DS

Regarding the recoloring, if there is any chance to find out the maximum file size depending on memory of instance, that would be useful to optimally use the recoloring features. I actually did some top commands while running but did not see any active processes using lots of memory.

[stackTom]

I have not fixed the flickering. Is the scrambling fixed though?

[stackTom]

Please git pull into insarmaps 3 and insarmaps 2 and delete the contents of the postgresql data directory (whichever one you specified into ./run_docker.sh). I had to change the postgresql version to 9.6 as it turns out we were using 9.6 and not 9.2. 9.2 does not have a feature I was using to speed up the database...

This postgresql version issues have been very frustrating. Now that we are on 9.6 they should disappear. Please ingest after pulling so that I can fix the flickering (unrelated to the database).

[stackTom]

I have fixed the flickering when the pixel size is changed.

I might have solved the scrambling on postgresql 16. I just need to ingest a dataset that was previously getting scrambled like S1_IW3_090_0622_0624_20170116_20240807_S08552_S08486_E123555_E123591_filtDel4DS. Please let me know where I can get this dataset so I can test.

[stackTom]

Great news. I have indeed figured out why it was scrambling on postgresql 16 and fixed it. An even nicer side effect of the fix is that it caused the recoloring to be faster on big datasets, please try it out.

So 1, 2, and 5 are fixed. As well as improvement in performance.

[falkamelung]

Hi @stackTom . This works fine now. Thank you. There is a small but but I will create a new issue on that. Closing this