GeoscienceAustralia / uncover-ml

Machine Learning system for Geoscience Australia uncover project
Apache License 2.0
30 stars 20 forks source link

training crashes when number of targets is low #49

Open basaks opened 5 years ago

basaks commented 5 years ago

When npartitions * nprocesses > ntargets, we have a bug.

930 Traceback (most recent call last):                                              
931   File "/g/data1a/ge3/john/venvs/class/bin/uncoverml", line 11, in <module>     
932     load_entry_point('uncover-ml', 'console_scripts', 'uncoverml')()            
933   File "/g/data1a/ge3/john/venvs/class/lib/python3.5/site-packages/click/core.py", line 764, in __call__
934     return self.main(*args, **kwargs)                                           
935   File "/g/data1a/ge3/john/venvs/class/lib/python3.5/site-packages/click/core.py", line 717, in main
936     rv = self.invoke(ctx)                                                       
937   File "/g/data1a/ge3/john/venvs/class/lib/python3.5/site-packages/click/core.py", line 1137, in invoke
938     return _process_result(sub_ctx.command.invoke(sub_ctx))                     
939   File "/g/data1a/ge3/john/venvs/class/lib/python3.5/site-packages/click/core.py", line 956, in invoke
940     return ctx.invoke(self.callback, **ctx.params)                              
941   File "/g/data1a/ge3/john/venvs/class/lib/python3.5/site-packages/click/core.py", line 555, in invoke
942     return callback(*args, **kwargs)                                            
943   File "/g/data1a/ge3/john/uncover-ml/uncoverml/scripts/uncoverml.py", line 57, in learn
944     targets_all, x_all = _load_data(config, partitions)                         
945   File "/g/data1a/ge3/john/uncover-ml/uncoverml/scripts/uncoverml.py", line 97, in _load_data
946     image_chunk_sets = ls.geoio.image_feature_sets(targets, config)             
947   File "/g/data1a/ge3/john/uncover-ml/uncoverml/geoio.py", line 427, in image_feature_sets
948     result = _iterate_sources(f, config)                                        
949   File "/g/data1a/ge3/john/uncover-ml/uncoverml/geoio.py", line 381, in _iterate_sources
950     x = f(image_source)                                                         
951   File "/g/data1a/ge3/john/uncover-ml/uncoverml/geoio.py", line 425, in f       
952     config.n_subchunks, config.patchsize)                                       
953   File "/g/data1a/ge3/john/uncover-ml/uncoverml/features.py", line 54, in extract_features
954     patchsize)                                                                  
955   File "/g/data1a/ge3/john/uncover-ml/uncoverml/features.py", line 34, in _extract_from_chunk
956     image_chunk = Image(image_source, chunk_index, total_chunks, patchsize)     
957   File "/g/data1a/ge3/john/uncover-ml/uncoverml/image.py", line 68, in __init__ 
958     nchunks, overlap)[chunk_idx]                                                
959   File "/g/data1a/ge3/john/uncover-ml/uncoverml/image.py", line 23, in construct_splits
960     p_min = s[0] - overlap