trouble saving mask sub cubes

[vdhulst]

In iDaVIE-v_v1.0-beta.6-preview8 the following problem occurs when saving sub cubes and corresponding mask cubes. When I load a cube of hgc44 produced by miriad and corresponding mask cube and then select a sub cube the following occurs when saving them:

• when I first save the sub cube mask using the menu and requesting a copy iDaVIE saves the full mask cube as a copy of the original mask cube, but not a sub cube.

• when I first save the (data) sub cube, then I do get a proper sub cube, but then the program gets stuck on producing the corresponding mask cube. It writes the proper fits header, but then keeps appending data until I stop iDaVIE. This way I was able to produce files uo to 29 GB in size (depending on how long I waited until killing iDaVIE) which are not readable.

However, when I repeat the above with a cube of the Fornax cluster, also produced by miriad these problems do not occur. The result from saving a sub cube is that both a sub cube and a mask sub cube are saved. This is confusing as one would expect that only a sub cube is saved and that for saving the mask cube one needs to push the sub cube button.

I propose to fix it such that the cube button saves a cube, the mask button saves a mask (if available of course) of the cropped region. If one wants to save a modified mask after paint and is uncropped, then it makes sense to save the entire mask cube. Now these options are mixed which is confusing.

[vdhulst]

When revisiting this problem I could not fully reproduce it. I get sub cube mask files for hgc44, but they are still not correct and only have zeros. The sub cube mask files for Fornax are ok.

In both cases when saving the sub cube it saves the sub cube and if available the corresponding mask. There are always two toaster messages, but when a mask sub cube is saved the toaster does not give the name of the mask sub cube. It gives the data sub cube name again. When saving the mask using the menu button iDaVIE always saves the full mask file.

When no mask is available the data sub cube is saved twice. When no mask is available it should just save one version of the data sub cube.

With the Fornax data, when I save a sub cube, I get an error message in the toaster saying error saving subcube (error # 105) when it saves the mask sub cube. The mask sub cube is, however, saved properly. For hgc44 I do not get this message, but the mask cubes are not readable.

[veggiesaurus]

@CosmicElysium we've assigned this to you. @vdhulst will go over steps to reproduce issue

[vdhulst]

@CosmicElysium: I ran a number of tests using iDaVIE-v_v1.0-beta.5 ad the advice of @veggiesaurus, so that we know what worked in that version and what did not work. I used four data sets: a cube of Fornax data (from @paoloserra), an Apertif cube (from @vdhulst), a cube of HGC44 (from @paoloserra) and a cube of Coma (from @paoloserra). I went through the following steps after loading the cube and corresponding mask: (i) select a subregion, save the subcube; (ii) crop, save the subcube: (iii) uncrop and again save a subcube.

In the case of the Apertif cube each step worked but produced two sub cubes and two mask sub cubes at each step. Each is correct and readable.

In the case of the Fornax cube I got 4 cubes (two identical data and two identical mask cubes) in each of step (i), (ii) and (iii). The sub cubes and the mask cubes are both correct and readable.

In the case of the Coma cube the situation is again different: each step produces a sub cube and a mask cube, but the mask cube is empty and in some cases can not be read. Occasionally iDaVIE would get stuck and produce and increasingly growing mask file, which in the end is unreadable, even though the header is OK.

in the case of the HGC44 cube the following happens: after step (i) iDaVIE gets stuck and needs to be restarted. It has produced one correct and one unreadable sub cube, no useable mask cubes and one mask cube just kept increasing in size until iDaVIE was restarted. Going to step (ii) immediately produces a sub cube and an unreadable maask cube. Uncropping and saving a sub cube again then causes iDaVIE to get stuck again.

Saving a copy of the original mask cube (other option in the menu) works fine for all cubes.

@CosmicElysium: I will provide the HGC44 cube and corresponding mask cube via Slack for testing.

[veggiesaurus]

@vdhulst thanks so much for the details. So to summarise, this is therefore an old problem, and not a regression in beta6?

[vdhulst]

@veggiesaurus: yes, that is what it looks like. I will check tomorrow whether I can reproduce this in beta6. In view of what I found originally this is almost certainly the case.

[CosmicElysium]

I will test and debug before Thursday

[vdhulst]

Tried the above again using beta6. Results are pretty much the same:

for the Apertif and Fornax cube: the same extra data and mask cube at each step, but both appear to be correct all the time.

For HGC44: got the runaway problem at step (i) and restarted iDaVIE. After restart step (i) and (ii) produce 4 files each, they are readable, the cubes are OK, the mask files are empty. Step (iii) again produces a runaway and needs to be stopped. Produces 4 files, one cube is readable and OK, one mask is readable but full of zero's (this btw is the file that keeps increasing in size, the other mask file is not readable).

Have not tried the Coma cube yet, but think that there is enough to debug using the HGC44 cube and above results. I noticed that beta5 is much faster in saving the sub cubes than beta6. So there is a subtle difference.

[vdhulst]

Did try the same on the system at the Kapteyn Institute, using beta6. Got the same results, so can confirm the various features are not system dependent.

[CosmicElysium]

The case with HGC44 seems to occur because the cube and mask have different dimensions. I cut out the fourth dimension of the cube to match the mask and exporting the subcube seems to work. This may be a straightforward fix, but I need to look a bit

[vdhulst]

yes I can confirm this. I made a new cube with three dimensions and then it works fine, except that instead of one data sub cube and one mask sub cube I get two of each.

Regarding the fourth axis I suggest that when sub cubes are saved, the 4th axis (if present) is stripped anyway as it is not relevant at all and not part of what the user 'sees' in iDaVIE. SoFiA does this too when making mask cubes, that is where the difference arose for HGC44 and Coma in the first place.

[CosmicElysium]

I think that is a good solution for subcubes to always export 3 dimensions. It will be a limitation that our software imposes, but idavie is a 3D data visualization tool, so it makes sense that it exports 3D cubes

[CosmicElysium]

@vdhulst I have not been able to recreate the two of each problem though. Do they have two different time stamps?

[vdhulst]

these are the files that were saved by iDaVIE after a select and crop on the 3-axes HGC44 cube:

hgc44_3axes_subCube_20220218_131339_mask.fits hgc44_3axes_subCube_20220218_131339.fits hgc44_3axes_subCube_20220218_131340_mask.fits hgc44_3axes_subCube_20220218_131340.fits

so yes, they do have different time stamps.

in the toaster I get two announcements that the data sub cube has been saved and the name of the data sub cube is shown. The first announcement (in practice the first set, data + mask) with 2 in the top right corner of the toaster window, the second announcement (i.e. set) with the name of the second data sub cube and with 1 in the top right corner. I have not managed to save only one set and get a toaster message with the name of the data and the name of the mask.

I also tried this in the latest version (main on GitHub, 2 day old now) and this still happens.

[CosmicElysium]

I am not getting the issue with subcube clones. Have tried a few times with source and build. Does it only happen with hgc44, @vdhulst ?

[vdhulst]

@CosmicElysium: no it happens with all cubes I tried so far, both at my laptop and at the system at the Kapteyn Institute. Maybe @paoloserra can confirm whether he has the same problem: i.e. one sub cube save gives two identical (except for the time stamp in the name) sub cube sets (data and mask). I tried with the formal beta6 release from 22 Feb and there I also get two sets of sub cubes (and two messages in the toaster).

[CosmicElysium]

I will try to recreate by exporting subcubes with the menu command. I have been using the voice command.

[vdhulst]

and I will test the voice command on my cubes.

[vdhulst]

I confirm that the problem does not arise when saving sub cubes using the voice command. Only the menu save sub cube option produces the clones.

[veggiesaurus]

I confirm that the problem does not arise when saving sub cubes using the voice command. Only the menu save sub cube option produces the clones.

Ok, so perhaps it's something to do with the actual UI. We might need to assign this to @fvitello ?

[CosmicElysium]

I can confirm that saving the subcube from the menu is getting some funny error when it works with the voice command for the same cube. It didn't do a double subcube for me, but something is definitely happening that should not.

[veggiesaurus]

@fvitello I've assigned this to you, can you look into why this is triggering twice?

[vdhulst]

maybe related to issue #65

[vdhulst]

tried saving sub cubes again with the current main branch from GitHub (https://github.com/idia-astro/idia_unity_vr). When using the voice command I get a proper sub-cube and corresponding sub-cube mask. When using the menu I get both as well, except that the toaster issues an error when saving the mask sub-cube: Error saving sub cube (error #105).

Re the menu driven sub-cube save: at the Institute I could continue doing things, on my laptop iDaVIE occasionally froze, but did write the files.

[Pietersielie]

@CosmicElysium Alex, follow up with Thijs on if this is still an issue and what causes it.

[vdhulst]

@Pietersielie and @CosmicElysium: I did check this issue using the latest prebuilt version "iDaVIE-v_v1.0-beta.9-preview1" and the problem is still there. Including the difference between using the voice command and the menu. So when using the menu there are two versions of the sub cube and corresponding mask, though with different time stamps. When using the voice command everything is fine, just one of each. No idea what causes this.

[vdhulst]

Just checked it with the latest master version from the GitHub repository. Also there the saving sub cubes problem is still there. So: voice command gives one set (sub cube + mask), the menu gives two sets, each set with its own timestamp. In that case the toaster also gives two messages, corresponding with each of the saved sets. The toaster messages are labelled 2 and 1.

[CosmicElysium]

@vdhulst I fixed the double saving issue. Could you please test the following build?

https://drive.google.com/file/d/1K-GrcYz4gEzCORNAHHb13xDCkJ2qiSyu/view?usp=drive_link