search

IBM / low-resource-text-classification-framework

Research framework for low resource text classification that allows the user to experiment with classification models and active learning strategies on a large number of sentence classification datasets, and to simulate real-world scenarios. The framework is easily expandable to new classification models, active learning strategies and datasets.
Apache License 2.0
97 stars 20 forks source link

Expected Gradient Length (EGL) not implemented? #2

Closed unre4l closed 3 years ago

unre4l commented 3 years ago

Hi, great framework. I've got a question though.

One AL strategy in your paper is Expected Gradient Length (EGL) but i can't find its implementation in the repo. I notice that the orchestrator_api.infer() method retuns a gradients key, but as far as i can see it is not utilised anywhere. Do you use the RetrospectiveLearner as an equivalent to EGL? What is the theoretical foundation?

arielge commented 3 years ago

Hi, Indeed, unfortunately we were unable to release the EGL implementation as part of the research framework (and this is unrelated to the RetrospectiveLearner). Of course, anyone who wishes is welcome to contribute to this or other parts of the repo.

unre4l commented 3 years ago

Thank you for the quick reply. This is indeed unfortunate as it narrows the comparability and reproducibility of your paper's results. Could there be more insight as to why it is not released along with the framework? I would be happy to know.

arielge commented 3 years ago

We had an early implementation for EGL, but we did not reimplement it for the open source as a) it was not better than other AL strategies despite being very inefficient b) returning per example gradients was not native to tensorflow and hence required a lot of effort to reimplement for a non-experimental version.