Python API UBODT.read_ubodt_binary() stops reading ubodt hash table

[davmol]

I'm encountering an issue where the hash table won't be read entirely via UBODT.read_ubodt_binary("/home/dmolitor/fmm/python/build/data/ubodt_deu_100_32632.bin"). It halts without any Error. The size of the .bin is 506 MB.

It's the latest fmm master clone on a fresh Cygwin install on Windows. Since there might be problems with the python path as mentioned here : https://github.com/cyang-kth/fmm/issues/71 import fmm statement can only be done within the ../python/build folder.

output:

XXX+dmolitor@XXX ~/fmm/python/build
$ python mm_test_2.py
[2020-05-15 15:34:24.124] [info] [network.cpp:37] Read network from file /home/dmolitor/fmm/python/build/data/network_deu_32632_6.shp
[2020-05-15 15:35:58.220] [info] [network.cpp:130] Number of edges 15997094 nodes 7119308
[2020-05-15 15:35:58.254] [info] [network.cpp:131] Field index: id 0 source 1 target 2
[2020-05-15 15:36:27.028] [info] [network.cpp:134] Read network done.
[2020-05-15 15:36:32.139] [info] [network_graph.cpp:17] Construct graph from network edges start
[2020-05-15 15:36:44.626] [info] [network_graph.cpp:30] Graph nodes 7119308 edges 15997094
[2020-05-15 15:36:44.766] [info] [network_graph.cpp:31] Construct graph from network edges end
[2020-05-15 15:36:45.100] [info] [ubodt.cpp:217] Reading UBODT file (binary format) from /home/dmolitor/fmm/python/build/data/ubodt_deu_100_32632.bin

[cyang-kth]

@davmol

Did you check run it directly in C++ program to see if the UBODT file can be read successfully?

[cyang-kth]

Also another issue is that the ubodt file is generated with the newest ubodt_gen or the older ubodt_gen_omp? A data type changes has been made so I am not sure if you rerun the ubodt_gen to generate that file or not.

[davmol]

I generated the hash table with the new ubodt_gen. Running the cmd line fmm with the trips.shp from the example as dummy gives me:

XXX+dmolitor@XXX ~/fmm/python/build/data
$ fmm --ubodt ubodt_deu_100_32632_6.bin --network network_deu_32632_6.shp --gps trips.shp
[info][fmm_app_config.cpp:49 ] Start reading FMM configuration from arguments
[info][fmm_app_config.cpp:78 ] Finish with reading FMM arg configuration
[info][fmm_app_config.cpp:98 ] ----   Print configuration    ----
[info][network_config.cpp:6  ] NetworkConfig
[info][network_config.cpp:7  ] File name: network_deu_32632_6.shp
[info][network_config.cpp:8  ] ID name: id
[info][network_config.cpp:9  ] Source name: source
[info][network_config.cpp:10 ] Target name: target
[info][gps_config.cpp:8  ] GPS format: GDAL trajectory
[info][gps_config.cpp:9  ] File name: trips.shp
[info][gps_config.cpp:10 ] ID name: id
[info][gps_config.cpp:11 ] Timestamp name: timestamp
[info][result_config.cpp:30 ] ResultConfig
[info][result_config.cpp:31 ] File:
[info][result_config.cpp:32 ] Fields: cpath mgeom
[info][fmm_algorithm.cpp:22 ] FMMAlgorithmConfig
[info][fmm_algorithm.cpp:23 ] k 8 radius 300 gps_error 50
[info][fmm_app_config.cpp:103] Log level 2-info
[info][fmm_app_config.cpp:104] Step 100
[info][fmm_app_config.cpp:105] Use omp false
[info][fmm_app_config.cpp:106] ---- Print configuration done ----
[info][network.cpp:37 ] Read network from file network_deu_32632_6.shp
[info][network.cpp:130] Number of edges 15997094 nodes 7119308
[info][network.cpp:131] Field index: id 0 source 1 target 2
[info][network.cpp:134] Read network done.
[info][network_graph.cpp:17 ] Construct graph from network edges start
[info][network_graph.cpp:30 ] Graph nodes 7119308 edges 15997094
[info][network_graph.cpp:31 ] Construct graph from network edges end
[info][ubodt.cpp:217] Reading UBODT file (binary format) from ubodt_deu_100_32632_6.bin
[info][ubodt.cpp:242] Read rows 1000000
[info][ubodt.cpp:242] Read rows 2000000
[info][ubodt.cpp:242] Read rows 3000000
[info][ubodt.cpp:242] Read rows 4000000
[info][ubodt.cpp:242] Read rows 5000000
[info][ubodt.cpp:242] Read rows 6000000
[info][ubodt.cpp:242] Read rows 7000000
[info][ubodt.cpp:242] Read rows 8000000
[info][ubodt.cpp:242] Read rows 9000000
[info][ubodt.cpp:242] Read rows 10000000
[info][ubodt.cpp:242] Read rows 11000000
[info][ubodt.cpp:242] Read rows 12000000
[info][ubodt.cpp:242] Read rows 13000000
[info][ubodt.cpp:242] Read rows 14000000
[info][ubodt.cpp:242] Read rows 15000000
[info][ubodt.cpp:242] Read rows 16000000
[info][ubodt.cpp:242] Read rows 17000000
[info][ubodt.cpp:242] Read rows 18000000
[info][ubodt.cpp:251] Finish reading UBODT with rows 18974878
[info][gps_reader.cpp:40 ] Read trajectory from file trips.shp
ERROR 4: Unable to open trips.shx or trips.SHX. Set SHAPE_RESTORE_SHX config option to YES to restore or create it.
[critical][gps_reader.cpp:45 ] Open data source fail

...finally it crashes before it notices that the .shp is humbug itself because of the missing the .shx. but that's okay. The hash table was read successfully!

[davmol]

Fresh install of cygwin with the newest libs resolved the issue.