HHS / simpler-grants-gov

https://simpler.grants.gov
Other
44 stars 13 forks source link

Create database table `ExtractMetadata` to store extract files #2791

Open mikehgrantsgov opened 5 days ago

mikehgrantsgov commented 5 days ago

Related to 2454

Summary

The legacy website provides the ability to download XML extracts of opportunity data: https://www.grants.gov/xml-extract - this is something we should also support. Having extracts (especially if they're just the same data as our search endpoints) reduces the traffic our search endpoint will need to handle as we can point anyone that would want to scrape our data to an extract file.

Create a table to track extract metadata.

Something like:


# Create LkExtractType as a lookup value with opportunities_xml and opportunities_csv as initial values

class ExtractMetadata(Base):
    __tablename__ = 'extract_metadata'

    extract_metadata_id = Column(BigInteger, primary_key=True, autoincrement=True)
    extract_type = Column(LkExtractType, nullable=False)
    file_name = Column(String, nullable=False)
    file_path = Column(String, nullable=False)
    file_size_bytes = Column(BigInteger, nullable=False)
    created_on = Column(DateTime, server_default=func.now(), nullable=False)
    updated_on = Column(DateTime, server_default=func.now(), onupdate=func.now(), nullable=False)

Acceptance criteria

chouinar commented 1 day ago

A few small suggestions:

mikehgrantsgov commented 1 day ago

Thanks, adapted this!