Perf improvements - Githubissues

CenterForOpenScience / SHARE

SHARE is building a free, open, data set about research and scholarly activities across their life cycle.

http://share-research.readthedocs.io/en/latest/index.html

Apache License 2.0

100 stars 58 forks source link

Perf improvements #770

Closed aaxelb closed 5 years ago

aaxelb commented 5 years ago

Investigating some memory/performance issues locally yielded a few improvements that might help:

Faster deduping algorithm for large input graphs
Skip parsing/deduping unrealistically long names (>200 characters), seems that nameparser sometimes chokes

Also allow the forceingest script more retries before giving up.

coveralls commented 5 years ago

Coverage increased (+0.03%) to 79.091% when pulling 5911259951fe4f62d453259e902a54286e0fa113 on aaxelb:fix/smoother-disambiguation into 86ab82ec100463614def6723d37e903b3b324985 on CenterForOpenScience:develop.