search

Klimatbyran / garbo

Klimatkollen's data pipeline, processing company sustainability reports
3 stars 6 forks source link

Check if the url is already parsed before continuing #124

Open schweinryder opened 4 months ago

schweinryder commented 4 months ago

If the url (pdf) is parsed and exist in the vector database. If it does: skip that part and fetch it from the db. Otherwise continue with the download and process of the pdf content as before.

irony commented 4 months ago

Perfect!