Open ghost opened 8 years ago
@epipremnum Yeah, that's a great idea. Just the last few days I have been working on "streaming" representations of corpora and papers (i.e. on disk, in a database) to cut down on memory overhead -- the logic is basically as you describe, to pass over the metadata records once and load into memory only the immediately-needed fields. So your suggestion is a logical continuation of that line of work. I'll keep this thread up to date as we work on it!
Thanks for using tethne
-- it would be good to hear more about your use-case, if you're willing to share. :-)
Hi Erick,
thanks for the quick response. That would be great.
For now i wrote a script to preprocess my bibliography files. I copied all lines corresponding to tags i wanted to keep to a new file. Tethne then gets only these lines as input that i am interested in. It works. But there is still a lot of memory used.
Thank you.
This will be TETHNE-124.
@epipremnum On the develop
branch I have added a parameter called parse_only
to the WoS and DfR readers. If you have a moment, it would be great to hear whether or not this addresses your need.
The relevant tests are here: https://github.com/diging/tethne/blob/develop/tethne/tests/test_readers_parseonly.py
This is now in v0.8.1.dev2, which can be installed via pip with --pre
:
$ pip install -U tethne --pre
Example:
>>> from tethne.readers.wos import read
>>> corpus = read('/path/to/my/data', parse_only=['title', 'date'])
Hi,
first of all, thanks for this amazing python package.
The corpus object in my analysis gets really huge. For example i don't need Funding information, mail adresses and some more. Is it possible to ignore certain Web of Science field tags when populating the paper-objects?
Like this?
ignore_tags = ['EM', 'FU']