Test data 'qm9.h5' and files could not be found

[ylyzz21]

Is your feature request related to a problem? Please describe.

Hello, Thank you so much for your effort for this repo. I have encountered such a problem at runtime, hope to get an answer.

When I run examples/gflownet/example_qm9.py to understand the use of gflownet, I found that there is no GFN/qm9.h5 file under this repo, which makes me unable to know what format the input data should be, and there is no way to carry out this case test.

Similar things happen when running RT, I can't find a tokenizer described in the sample code, I think I need an example of this, to know how to create my own tokenizer. gt4sd-trainer --training_pipeline_name regression-transformer-trainer --tokenizer_name ~/.gt4sd/algorithms/conditional_generation/RegressionTransformer/RegressionTransformerMolecules/qed --config_name examples/regression_transformer/rt_config.json --do_train --output_dir my_regression_transformer --train_data_path src/gt4sd/training_pipelines/tests/regression_transformer_raw.csv --test_data_path src/gt4sd/training_pipelines/tests/regression_transformer_raw.csv --overwrite_output_dir --eval_steps 200 --augment 5 --eval_accumulation_steps 1

Describe the solution you'd like I really hope to see the data directory under the gt4sd-core folder, which contains all the data files mentioned in the sample code, especially the qm9.h5 and ~/.gt4sd/algorithms/conditional_generation/RegressionTransformer/RegressionTransformerMolecules/qed tokenizer

Describe alternatives you've considered I thought about inverting a data hierarchy by using the h5 file described in dataset.py, but this seems too stupid. So I still hope that the authors can provide a corresponding file directly, saving us too many detours.

Additional context Thanks again.

[jannisborn]

Hi @ylyzz21, Thanks for the interest in the repo. In general, GT4SD uses a cloud object storage to store artifacts such as pretrained models or tokenizers, similar to HuggingFace. Once you execute a certain model, a download is triggered and the artifacts are synced to your local cache which defaults to ~/.gt4sd.

Regarding your specific problems:

• RT: You cant find a tokenizer locally as long as you never used any RT model locally. You can trigger the caching for any of the RT models like described here: https://github.com/GT4SD/gt4sd-core/tree/main/examples/regression_transformer. Note that for any RT training, we recommend to warm-start from one of the ~10 available pretrained models. Maybe you can describe your usecase / data formatting so that we can guide you on the tokenizer/model to use for initialization. Also note that, the RT does not support general tokenization schemes, you will have to stick to one of these tokenizers listed in this if/elif clause: https://github.com/IBM/regression-transformer/blob/e0a5e52b45739c975754d6014b223458e3aad26d/terminator/tokenization.py#L261 or submit a PR there with your custom tokenizer. If you just want to substitute the vocab but keep the tokenizer, this of course should work out of the box!
• GFlowNet, maybe @georgosgeorgos can comment here since he wrote this GFlowNet implementation

[jannisborn]

Closing the issue @ylyzz21, let us know if you need further information