Accessing rows from the HuggingFace data_loader with the "csv" loader seemed to be slowing down the MTR training. Made some changes that should fix this. The bigger the dataset, the more of an improvement this should make. With 1k compounds you cannot tell the difference between the old + new methods, with 1M compounds we do 160% more iterations/s. Haven't benchmarked with larger datasets since I'm doing this on a small machine but we should test this tomorrow with the full dataset.
What's in the PR:
the benchmarking notebook has the work that shows how much slower it is to load a row of pubchem data from the "csv" loader compared to the "text" loader
Based on this plot ^ wrote RegressionTextDataset that treats the csv as a text object and just reads each line in then during processing splits the string into a list of floats
Switched the default regression dataset to be a RegressionTextDataset
Removed the annoying statement that kept printing the # rows collating so the training progress tracker is a lot easier to track
This makes a few assumptions (that are easy to stick with):
None of our CSVs have headers
Removed the header from the 1k default test file so that it runs
The first column is the smiles column, the rest are floats
Accessing rows from the HuggingFace data_loader with the "csv" loader seemed to be slowing down the MTR training. Made some changes that should fix this. The bigger the dataset, the more of an improvement this should make. With 1k compounds you cannot tell the difference between the old + new methods, with 1M compounds we do 160% more iterations/s. Haven't benchmarked with larger datasets since I'm doing this on a small machine but we should test this tomorrow with the full dataset.
What's in the PR:
RegressionTextDatasetthat treats the csv as a text object and just reads each line in then during processing splits the string into a list of floatsThis makes a few assumptions (that are easy to stick with):