search

Altinn / digdir-slack-bot

MIT License
2 stars 0 forks source link

[scraper] run scraper as scheduled job or for each push to master #61

Closed bdb-dd closed 10 months ago

bdb-dd commented 11 months ago

Description

We should automatically update the altinn-studio-docs collection in typesense by running the scraper.

Since adding the embedding column to the collection, job time has increased to several hours.

Would incremental sync be an option? Should be easy to infer which URLs need re-indexing based on files included in commit.

Additional Information

No response

Tasks

No response

Acceptance Criterias

No response

bdb-dd commented 10 months ago

replaced by #70 Scraper job time is fast again now that we have externalized the embedding generation.