So, there are occasionally amendments to disclosure reports. These are also posted right alongside the reports and, good news, because they are formatted the same as the other disclosure pages our scraper is pulling and processing them correctly. The bad news is, they violate my current primary key rule of one unique entity+date_range in the headers table, and therefore EVEN THOUGH they have good good data they aren't getting saved properly. I should switch to using the URL as the primary key for the headers table rather than the current compound key.
KaylaCrush
commented
1 year ago
So, there are occasionally amendments to disclosure reports. These are also posted right alongside the reports and, good news, because they are formatted the same as the other disclosure pages our scraper is pulling and processing them correctly. The bad news is, they violate my current primary key rule of one unique entity+date_range in the headers table, and therefore EVEN THOUGH they have good good data they aren't getting saved properly. I should switch to using the URL as the primary key for the headers table rather than the current compound key.
See https://www.sec.state.ma.us/LobbyistPublicSearch/Summary.aspx?sysvalue=2dqyBaVpsazd/6dZu3fpZlh9S9KxjUElI+HvUTjnKHo=