search

noaa-ocs-hydrography / kluster

A distributed multibeam processing system built using the Pangeo ecosystem
Creative Commons Zero v1.0 Universal
47 stars 12 forks source link

Error redrawing 3d points after filtering #91

Open davesteps opened 2 years ago

davesteps commented 2 years ago

Hi @ericgyounkin, This happens intermittently. I am trying to pin down the conditions under which it happens:

I subset some data like so: image

I apply a filter, in this case the angle filter -25+25. When the filter finished the points are redrawn like so: image

However if I reselect the subset, the data is correct, indicating that the data written to disk is correct. image

As I said it does not happen all the time. It happens with other filters so not related to the angle filter. It does seem to be related to having multiple lines selected as I don't think it happens if all the data is from one line. I will keep trying to get insight into the problem. Thanks.

davesteps commented 2 years ago

^ Ignore the point about multiple lines. I have filtered multiple lines without seeing the error

ericgyounkin commented 2 years ago

@davesteps is it only on the new dev branch that you see this? Or did you also see it before.

davesteps commented 2 years ago

I didn't notice it previously, but I have just tried and I can recreate it on master also

davesteps commented 2 years ago

It can be reproduced with the 0009_20170523_181119_FA2806.all test data also. Selecting the entire set to view in 3d points view:

image

ericgyounkin commented 2 years ago

oh strange. OK, that is very helpful, ill take a look at it today.

ericgyounkin commented 2 years ago

@davesteps Can you tell me more about the steps you took to reproduce the issue with 0009_20170523_181119_FA2806.all? It looks it might have been filtered once already, and then you filtered again? Not that there is anything wrong with that, just trying to reproduce on my machine.

My processed version of that line does not match your screenshot, so I assume you filtered once.

image

davesteps commented 2 years ago

I've just tried again from scratch (this is on master): 1) load in the .all file and process 2) build a surface 3) Select all: image 4) color to rejected: image 5) Filter by angle (use defaults +-45): image Results in: image 6) redraw subset: image

davesteps commented 2 years ago

If you do not get the same results then it might be down to my environment. Perhaps with installing our packages something as changed and I'm using different versions to you.

ericgyounkin commented 2 years ago

@davesteps Can't reproduce. But, there is an underlying assumption to the filter -> point view update that could be wrong. I assume that the order the points are loaded into the points view is the same as the order that the filter results are in, in terms of (time, beam). This might be a bad assumption, for some reason that you have encountered that i do not understand. It is certainly a weak assumption, so a fix is probably a good idea regardless.

Either way, I can just throw in another sorting step to match (time,beam) of filter results to points view. I added a fix to the dev branch. That should do it.

https://github.com/noaa-ocs-hydrography/kluster/commit/424b27b8976b09fd1220670ea01be931856d9ead

ericgyounkin commented 2 years ago

There was one last potential issue I could find, which is where the points view points are not ordered in time, beam. This should not happen, but just in case, I put in another sort which should align the new status update regardless.

https://github.com/noaa-ocs-hydrography/kluster/commit/3faec83293afb27a283c27b8678cdee630250995

davesteps commented 2 years ago

Hi @ericgyounkin, It's strange but I'm still getting it. The fact that you cannot recreate the behaviour at all makes me think it must be something to do with my env. I'll build a new one and test.

ericgyounkin commented 2 years ago

@davesteps OK, although I can't think of what would cause the difference. We have different OS, but I don't see that being the cause.

I should mention that gridding is going to be probably somewhat broken on the dev branch right now. I'm in the process of adding/testing the CUBE algorithm in gridding.

davesteps commented 2 years ago

Hi @ericgyounkin, so I have started from scratch with a new environment and weirdly still get the behavior. Using Ubuntu 20.04 on master (tested on dev also) using the steps below to setup and then the processing the data as described above. 

conda env create -n kluster_test2 -f environment.yml
conda activate kluster_test2
pip install -e .
mkdir /home/david/miniconda3/envs/kluster_test2/share/qgis/plugins
python -m HSTB.kluster

note the mkdir step, it requires that the qgis plugins folder exists even if empty

ericgyounkin commented 2 years ago

@davesteps OK, I updated my ubuntu VirtualBox and I am able to replicate it there. I can resolve that plugins directory thing easy, it's just in a different place in linux for some reason.

The filter update bug is so weird. It only seems to present if you select the whole dataset. If I get just a slice of the dataset, the update seems to work. Is that what you are seeing too?

davesteps commented 2 years ago

Great that you are seeing it too ;) On the 0009_20170523_181119_FA2806.all example yes it is only when the whole data is selected. On the other example it also happens with subsets of data. It is not easy to tell exactly when it happens but if I only have a small subset on one side of line it doesn't tend to happen, however larger subsets over the entire width of the line it is more likely to happen.

ericgyounkin commented 2 years ago

@davesteps This is the weirdest thing I think I have come across yet in my programming work. I resolved the path issues that I found that were linux specific:

https://github.com/noaa-ocs-hydrography/kluster/commit/f8d106a579bedec6c7aadc9e66c0d56a05eb277f

And the issue is gone. At least in my testing on my ubuntu box. I don't understand why. Can you test the new dev branch to make sure I'm not crazy?

davesteps commented 2 years ago

Hi @ericgyounkin I tested the latest commit on dev and am still seeing it

ericgyounkin commented 2 years ago

@davesteps Sorry, this is kind of frustrating. I still am having issues reproducing on my Ubuntu VM. I released kluster 0.9.1 yesterday, I was wondering if you could switch to that branch and try. Might be good to share your 'help - about' screen too, just to ensure the dependency versions match between our environments.

image

davesteps commented 2 years ago

Hi @ericgyounkin here are my versions: image

davesteps commented 2 years ago

Hi @ericgyounkin,

I don't know if this might shed some light. On another dataset the points don't redraw and it returns this error:

Traceback (most recent call last):
  File "/home/david/kluster/HSTB/kluster/gui/kluster_main.py", line 1336, in _kluster_filter_results
    base_points_view_status[matches_sonar] = ninfo[fqheadsel][results_indices]
IndexError: index 44252 is out of bounds for axis 0 with size 44252
Selected 47166 Points for display

Upon redrawing the subset the data looks correct as before

ericgyounkin commented 2 years ago

Yeah, I really don't understand how that could be. I was only able to reproduce this once on my virtual box ubuntu instance. I need to get back to it, I just have a few things I gotta deal with first.

Do you think you could catch this in a Pycharm debugger? Or whichever IDE your team uses? If we could organize another Teams session so that I could see it, maybe that would help. Just a thought.

davesteps commented 2 years ago

I have tried but the debugger can't step into the filter method for some reason, something to do with how they are being instantiated/called I think, I'll try again. Happy to have a call again if it helps.

ericgyounkin commented 2 years ago

Yes, you will not be able to step into the Filter as it is run in a separate thread from the main GUI. You can step in to the filter results though, thats probably where we should start. I can email about meeting up, after I finish this other thing.