iDigBio / idb-backend

iDigBio server and backend code for data ingestion, media processing, record indexing, and data API.
GNU General Public License v3.0
7 stars 0 forks source link

updates for neherbaria rss and mediaing #236

Closed jbennettufl closed 6 months ago

jbennettufl commented 6 months ago

This updates the RSS script so that we may use it against RSS feeds which were not able to be viewed before due to UTF-8 encoding errors and changes update_publisher_recordset so that only the "link" field is formatted with UTF-8. It is a good place to start if we need any modifications for problematic RSS feeds. Currently, if any other fields are not valid UTF-8 they are not changed. MediaING process has been updated to accept a date range. So far this has proven to work with Neherbaria and will also be necessary to perform regular updates via ingestion without having to run it against the entire database or just for a 24 hour period.