Closed adamfarquhar closed 2 years ago
We can patch a new_empty
method to our dataloaders inside convert_dataloader_to_fastai.py
, what do we need to return in this function?
I am not completely sure, but I think it should return a copy of the dataloader with an empty dataset. I.e., it is empty except that it retains the transforms. I guess that the idea is that the learner is being exported for inference. It should include the model and a variety of params, but it should not include the data that was present in the training environment. This is implemented by creating dataloaders that are empty of data.
Even if we temporarily define null values of dataset.new_empty (as lambda: None), we stumble on the inner function splitter in the fastai adaptor. The inner function can't be pickled using the default module. In fastai 2.1.4, the the pickle_module is not available as an argument. This has subsequently been updated. But then providing 'pickle_module=dill' stumbles on
PicklingError: Can't pickle typing.AbstractContextManager: it's not found as typing.AbstractContextManager
.
So, the best solution will be to define a proper new_empty and then eliminate the inner functions (which don't in this case add any value and could be replaced with regular functions).
Hi everyone.. there is still no solution for this at the moment right? What would be your recommendation as a workaround? Thanks!
@ai-fast-track do we need this one at all? I believe you have figured out the saving+reloading model thing.
Closing as saving+reloading is taken care of here.
🐛 Bug
Describe the bug The fastai leaner's export routine requires a
new_empty()
method on the dataset (doc: Create a new empty version of the self, keeping only the transforms). The fastai implementation accesses the fastai dataset'stls
attribute (transformed lists), which has anew_empty()
method.Not sure how best to fix this in icevision. The icevision dataloader has a
new
method, but this retains the records.To Reproduce Create a fastai learner with any icevision model and dataset. Try to
learn.export()
.Expected behavior
learn.export()
should save the model without data.Additional context