Fix moderation and add pinecone update with hash ids

StampyAI / alignment-research-dataset

Stampy's copy of Alignment Research Dataset scraper

MIT License

9 stars 7 forks source link

Closed henri123lemoine closed 1 year ago

henri123lemoine commented 1 year ago

This PR fixes the moderation problem where there could only be 32 chunks moderated per call.
It also fixes pinecone_update_required stuff, where now it will only be set to True if any of the PineconeMetadata attributes have been modified and is_valid is true.

Sorry for again including some cosmetic changes in here, but you guys need to pull from main for your commits to auto-include running black.

henri123lemoine commented 1 year ago

The error occurs for unknown reasons. I do not have this error locally when I run pytest.