search

graphnet-team / graphnet

A Deep learning library for neutrino telescopes
https://graphnet-team.github.io/graphnet/
Apache License 2.0
85 stars 86 forks source link

I3 converter bugs #662

Closed Aske-Rosted closed 4 months ago

Aske-Rosted commented 5 months ago

I encountered two bugs when trying to create files with the new general dataconverter.

  1. The output_files list was instantiated in a way where the folder for the output files where checked for existing files before these had been created.
  2. The _create_file_name were created in a way that did not allow for multiple "." to be in the input file name (which is often the case for our I3 files).
  3. The create_table_and_save_to_sql in _merge_databases for the SQLiteWriter was instantiated without a default_type, which means that it defaults to NOT NULL effectively not allowing for missing values when merging the databases.

This pull request should fix these issues.

RasmusOrsoe commented 4 months ago

@Aske-Rosted Thanks for sharing this. I noticed that there are a few unit tests that are not passing. Could you take a look at those?

Aske-Rosted commented 4 months ago

@Aske-Rosted Thanks for sharing this. I noticed that there are a few unit tests that are not passing. Could you take a look at those?

I have tried to take a look at them they are all about "ModuleNotFoundError: No module named 'torch_geometric.utils.homophily'" which is not a module I have introduced in this PR so I do not really understand why these are showing up here...

RasmusOrsoe commented 4 months ago

@Aske-Rosted I've pushed a fix to the import error here #663

Aske-Rosted commented 4 months ago

I encountered two bugs when trying to create files with the new general dataconverter.

1. The `output_files` list was instantiated in a way where the folder for the output files where checked for existing files before these had been created.

2. The `_create_file_name` were created in a way that did not allow for multiple "." to be in the input file name (which is often the case for our I3 files).

3. The `create_table_and_save_to_sql` in ` _merge_databases` for the `SQLiteWriter` was instantiated without a `default_type`, which means that it defaults to `NOT NULL` effectively not allowing for missing values when merging the databases.

This pull request should fix these issues.

I found one more issue for which I implemented a solution. The Issue was that using the Generic extractor to try and extract a value from an I3PODHolder (https://software.icecube.wisc.edu/icetray/main/projects/icetray/i3podholder.html) would not be extracted properly. The data would per table be extracted as a python dict with with the key being "value". The solution I have implemented catches this exact instance and extracts the value of the dict.