GeoscienceAustralia / uncover-ml

Machine Learning system for Geoscience Australia uncover project
Apache License 2.0
30 stars 20 forks source link

AttributeError: 'NoneType' object has no attribute 'predict' - possible multiprocessing problem - Windows 10?? #124

Open bluetyson opened 3 years ago

bluetyson commented 3 years ago

The problem occurs when doing OOS validation and multiprocessing by the looks. A .model file is created and looks fine when I check it... as in, I loaded one created and it had right number of feature importances, not tried prediction. Same process/data worked on out of sample when not run multiprocessing. Model exists when checked in predict.py - kwargs and RandomForestTransformed get returned ok - and that branch clearly works fine in general - I have run quite a lot of multiprocessing regressions with no OOS section with no problem.

+59s INFO uncoverml.scripts.learn_cli [P0]: Learning full randomforest model
+83s INFO uncoverml.validate [P0]: Performing out-of-sample validation with 5301 targets...
Uncaught exception on processor 2
Traceback (most recent call last):
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\gdaltest\Scripts\uncoverml.exe\__main__.py", line 7, in <module>
    sys.exit(cli())
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)

  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\scripts\__init__.py", line 142, in learn
    learn_cli.main(config_file, partitions)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\scripts\learn_cli.py", line 64, in main
    oos_results = ls.validate.out_of_sample_validation(model, oos_targets, oos_features, config)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\validate.py", line 349, in out_of_sample_validation
    lon_lat=pos)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\predict.py", line 56, in predict
    result = apply_masked(pred, data)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\models.py", line 1113, in apply_masked
    res = func(mr.trim_mask(data), *args, **kwargs)
  File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\predict.py", line 41, in pred
    predres = np.reshape(model.predict(X, **kwargs), newshape=(len(X), 1))
AttributeError: 'NoneType' object has no attribute 'predict'
bluetyson commented 3 years ago

Basic config when encountered the problem :-

mpiexec -n 3 uncoverml learn N032Dummy.yaml

[same error if a percentage OOS or a separate shapefile]

learning:
  algorithm: randomforest
  arguments:
    n_estimators: 50
    target_transform: standardise
    random_state: 42

validation:
  out_of_sample:
    percentage: 0.2
    shapefile: smalloutofsample.shp
    property: target
  k-fold:
    folds: 3
    parallel: True
    random_seed: 42

features:
  - type: ordinal
    files:
    - directory: F:\Modelling
    transforms:
      - centre
      - standardise
    imputation: mean

targets:
    file: F:/targetsmain.shp
    property: target

mask:
  file: F:/mask.tif
  retain: 1

prediction:
  quantiles: 0.95
  outbands: 4

output:
  #plot_feature_ranks: True
  #plot_intersection: True
  #plot_real_vs_pred: True
  #plot_correlation: True
  #plot_target_scaling: True
  directory: F:/RESULTS/