[NLP]very slow processing for certain posts

Common-SenseMakers / sensemakers

Sensemakers infrastructure for developing AI-based tools for semantic annotations of social posts. Cross-poster app to publish your semantic posts on different networks.

GNU General Public License v3.0

1 stars 2 forks source link

[NLP]very slow processing for certain posts #27

Open ronentk opened 5 months ago

ronentk commented 5 months ago

Example: https://fediscience.org/@petersuber/112083114047048300

Issue seems to be in url normalization https://github.com/Common-SenseMakers/sensemakers/blob/e31297e349ac1280bb36d7b6f5ec1fb63ecc36d7/nlp/desci_sense/shared_functions/dataloaders/mastodon/mastodon_utils.py#L92