Closed kYwzor closed 5 years ago
At least the current way of handling sparse matrix in libact.base.dataset
is not ideal.
def __init__(self, X=None, y=None):
if X is None: X = []
if y is None: y = []
self.data = list(zip(X, y))
self.modified = True
self._update_callback = set()
list(zip( X,y ))
would create a list of sparse feature vector and label pairs.
When evoking method get_labeled_entries()
, it is just apply a filter to self.data
and the output format is not acceptable to scikit-learn fit
interface.
If using format_sklearn
, it is transforming into a numpy array with rows of scipy sparse matrix. This would not be accepted by scikit-learn either. A correct way is to stack them
Yes, currently we don't support sparse feature vector, it should be a documentation error.
Looks like the main reason is caused by get_labeled_entries
and get_unlabeled_entries
methods in libact.base.dataset.Dataset
. Is there any specific reason for returning a single list instead of returning a tuple of feature matrix and list of labels?
I searched the package and saw most of the usage of these two methods are eventually zipping the output, making them into a matrix and list. I don't see why zipping them after getting them instead of changing the interface.
I think it is handled by #165.
I'm trying to implement Active Learning on sentiment analysis of sentences using a bag-of-words model. To get the model I'm using scikit-learn's CountVectorizer, which outputs a SciPy csr_matrix (a sparse matrix with token counts). Below is an excerpt of my code, which has df as a Pandas dataframe, with df.text being the sentences and df.sentiment their corresponding labels (3 classes).
According to libact's docs, the LogisticRegression model interfaces scikit-learn’s logistic regression model, which claims to be able to "handle both dense and sparse input". As such, I believe this is unexpected behaviour, If the code is changed as follows...
... the rest of the program runs perfectly. However, since my dataset and vocabulary are fairly large (over 65k sentences), I'm running into a lot of out of memory problems, which I believe would be fixed if I could use sparse matrices. I'm not sure what's the root of the issue exactly, but I'm suspicious of the format_sklearn() method, since there's quite a lot of conversions going on there. However, I wasn't able to investigate it thoroughly, so I can't pinpoint the problem yet.