fastr - found HDF5, which is not found in the typelist

[BartWitjes]

When I use fastr for the trainclassifier function, I get the following error:

File "", line 1, in trainclassifier.trainclassifier(test, patientinfo, config, output_hdf, output_json)

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\PREDICT\trainclassifier.py", line 226, in trainclassifier tempsave=config['General']['tempsave'])

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\PREDICT\classification\crossval.py", line 295, in crossval **config['HyperOptimization'])

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\PREDICT\classification\parameter_optimization.py", line 82, in random_search_parameters random_search.fit(features, labels)

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\PREDICT\processing\SearchCV.py", line 1628, in fit return self._fit(X, y, groups, sampled_params)

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\PREDICT\processing\SearchCV.py", line 1319, in _fit estimator_data = network.createsource('HDF5', id='estimator_source')

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\fastr\core\network.py", line 543, in create_source sourcenode = SourceNode(datatype=datatype, id=id_, parent=self, nodegroup=nodegroup)

File "C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\fastr\core\node.py", line 1243, in init raise exceptions.FastrValueError(message)

FastrValueError: [fastr:///networks/PREDICT_GridSearch_E7F5941TN9/0.0/nodelist/estimator_source] Unknown DataType for SourceNode fastr:///networks/PREDICT_GridSearch_E7F5941TN9/0.0/nodelist/estimator_source (found HDF5, which is not found in the typelist)!

There was no config file in my $HOME/.fastr/ folder, but also after putting it there ($HOME/.fastr/config.d/PREDICT_config.py), I got the same error.

[MStarmans91]

When using fastr, all data has to have a specific type, so it can check whether sinks, sources and nodes that are connected match. These types have to be in defined in fastr, see also https://fastr.readthedocs.io/en/stable/static/user_manual.html#datatypes .

In WORC, we have added several datatypes. Although the error you get is in PREDICT, it needs a datatype we defined in WORC (yes, that's a bit inconvenient). Hence, you need to tell fastr where these files are located. They are in a specific folder in WORC: https://github.com/MStarmans91/WORC/tree/master/WORC/resources/fastr_types . You can also see the HDF5.py there, which fastr apparantly cannot find. To tell fastr to look here, WORC should install a configuration file into the fastr config folder. I guess that one is missing.

Your fastr config folder is in your home folder: $HOME/.fastr, which is hidden folder. In there, there could be a general config.py. Also, there should be a config.d folder in there, where the WORC and PREDICT configs should be located.

Can you try putting the WORC config (https://github.com/MStarmans91/WORC/tree/master/WORC/fastrconfig) also in the config.d folder? You can check whether fastr picks this up by importing fastr and printing fastr.config.types_path. The installation folders of both WORC and PREDICT should be in there.

Let me know when you put the config there what the output of fastr.config.types_path is, so we can check if at least the config is parsed correctly.

[BartWitjes]

Thanks! I put both PREDICT and WORC config in the config.d folder and this is the output for fastr.config.types_path:

['C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\fastr\resources\datatypes']

[MStarmans91]

Apparently, these files in the config.d folder are not parsed by fastr. Otherwise, 'C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages\WORC\resources\fastr_types' would also have been in that list.

Just a quick check: do you have fastr>=2.0.0? The config.d feature was introduced in 2.0.0. You should have, as WORC requires this, but just to be sure.

This is more a fastr issue than a WORC issue at this point. I guess fastr is not parsing this folder at all. You could try to pragmatically solve this by putting a config.py in the .fastr folder and putting the content from both the PREDICT_config.py and the WORC_config.py files in there. But maybe we can check where fastr is looking for your config, as it can be in multiple locations, see https://fastr.readthedocs.io/en/stable/static/file_description.html#config-file.

Can you check whether you have a different fastr home folder? Just open a command window (Windows+R button, type cmd'' and press enter) and runecho $FASTRHOME''. If that is defined, than you have to define a config.d folder in there and put the WORC and PREDICT configs in there.

Hope that one of those options solves the issue.

[BartWitjes]

Yes, I have fastr version 2.1.1.

I tried putting the content of both into one config file, but that didn't work.

I'm not sure what you mean with the second option; when I type echo $FASTRHOME, it just returns '$FASTRHOME'?

[MStarmans91]

Sorry, on Windows I guess you have to do echo %FASTRHOME% .

If that returns %FASTRHOME% as well, than the variable does not exist. In that case, fastr should use the ~/.fastr folder to detect the config and we have to look for a different issue.

[BartWitjes]

Ok, unfortunately this also returns %FASTRHOME%.

But I did just notice that there are additional lines to the error, since I put the config files there:

[WARNING] basemanager:0287 >> Cannot scan C:\Users\Gebruiker\Anaconda3\envs\py27\WORC\resources\fastr_tools with ToolManager, path does not exist! [WARNING] basemanager:0287 >> Cannot scan C:\Users\Gebruiker\Anaconda3\envs\py27\PREDICT\fastr_tools with ToolManager, path does not exist! [WARNING] basemanager:0287 >> Cannot scan C:\Users\Gebruiker\Anaconda3\envs\py27\PREDICT\fastr_tools with ToolManager, path does not exist! [WARNING] basemanager:0287 >> Cannot scan C:\Users\Gebruiker\Anaconda3\envs\py27\WORC\resources\fastr_types with DataTypeManager, path does not exist! [CRITICAL] node:1242 >> Unknown DataType for SourceNode fastr:///networks/PREDICT_GridSearch_OY24WX42B8/0.0/nodelist/estimator_source (found HDF5, which is not found in the typelist)!

I think it should be looking in C:\Users\Gebruiker\Anaconda3\envs\py27\Lib\site-packages? How can I change this?

[MStarmans91]

Ok, that sheds a different light on the issue. Fastr does parse your configs in the config.d, but the paths on which it should look are indeed incorrect. If you look into the WORC_config.py and PREDICT_config.py, you see that we automatically try to locate the packagedir, which indeed in your case will probably be C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages as PREDICT is installed in there.

Probably this is because you are on Windows. I will try to find a method that works on Windows: for now, in those configs, just define the packagedir manually before tools_path and types_path are altered, e.g. for the PREDICT_config.py:

packagedir = 'C:\Users\Gebruiker\Anaconda3\envs\py27\lib\site-packages' 
tools_path = [os.path.join(packagedir, 'PREDICT', 'fastr_tools')] + tools_path

[BartWitjes]

Yes it works now! Thanks a lot!!