fecgov / openFEC

The first RESTful API for the Federal Election Commission. We're aiming to make campaign finance more accessible for journalists, academics, developers, and other transparency seekers.
https://api.open.fec.gov/developers
Other
480 stars 106 forks source link

Refactor Elasticsearch management functions #4399

Closed lbeaufort closed 1 year ago

lbeaufort commented 4 years ago

Summary In the past, when citations or any entity were missing for a specific AO's or a Case (MUR, ADR, AF) on the website, we run the zero downtime reload command which upload all the legal documents onto elasticsearch service. The Zero downtime task takes ~1.5 hours without causing any downtime on the website and uploads all legal documents. With the regular initialize load commands (for each document type) we anticipate downtime.

Things to consider:

Completion criteria:

pkfec commented 3 years ago

Two tasks that we run to upload documents onto elasticsearch service after which the legal documents are available on the website for legal search:

  1. Zero downtime - Causes no downtime. Loads all the docs and take ~1.5 hours.
  2. Initialize docs - Its real time process. Causes downtime until each document is uploaded on the elasticsearch service.

    It appears that we have all of our legal documents uploaded onto one elasticsearch index. After discussing with @fec-jli @lbeaufort we want to create a separate index for each document type and write a zero downtime task for each one of the document type so that we can speed up the upload documents without causing any downtime on the website.

pkfec commented 3 years ago

Un assigning myself as I am not actively working on this issue.