`Pipeline` does fails when setting `memory`

basf / MolPipeline

MIT License

89 stars 4 forks source link

`Pipeline` does fails when setting `memory` #51

Closed c-w-feldmann closed 1 month ago

c-w-feldmann commented 1 month ago

Currently this feature is not supported, as the cached transformers are not properly set after loading the cached results.

Todos:

[x] find underlying cause
[x] fix bug
[x] refactor
[x] Write unit tests

c-w-feldmann commented 1 month ago

~~Currently a _MolPipeline object is create dynamically, thus each run it has a new ID. This prevents joblib from loading cache (I suspect)~~

Edit
It uses hashing, and seems to work. Only downside is, if the Pipeline contains only one Element, or all Elements can be aggregated to one subpipeline, no caching is done.