169 pull regulation comments 5k not fully integrated

[GregoryHo88]

As discussed,

1.) Removed commentOnID as our cronjobs pull using start and end dates 2.) Extended >5k functionality to documents

[andrewjtdunn]

One other instance that I think raises a bug -- when I'm trying to search via commentOnId, I don't think the params quite work:

`object_id = '09000064864eba2d'

comment_data = pull_reg_gov_data( REG_GOV_API_KEY, "comments", params={"filter[commentOnId]": object_id}, )`

starts fetching data from 2006

[GregoryHo88]

This seems to work for documents and comments, and the print statements are helpful for seeing what's going on.

The code doesn't seem to work when I try to add dockets to the database using move_data_from_api_to_database.py. I get this error: requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.regulations.gov/v4/dockets?filter%5BsearchTerm%5D=CMS-2024-0158&page%5Bsize%5D=250&sort=lastModifiedDate%2CdocumentId. Is it possible that 'params' isn't updating?

I have resolved the issue for dockets. It is working now.

You were right about the how the params wasn't updating properly, the issue was in the if statement.

[GregoryHo88]

One other instance that I think raises a bug -- when I'm trying to search via commentOnId, I don't think the params quite work:

`object_id = '09000064864eba2d'

comment_data = pull_reg_gov_data( REG_GOV_API_KEY, "comments", params={"filter[commentOnId]": object_id}, )`

starts fetching data from 2006

On this issue, the former version of the code chains API requests based on commentOnId, I have since removed this based on our discussion on Monday, but do let me know if you would want this feature back.

I feel that we are kinda implementing too many extensions on pull_reg_gov_data(). I like your idea on refactoring this into its own data class with each method making a different class of API calls.

[andrewjtdunn]

FYI I merged the updated main into the branch to capture the add_comments_to_date_range function in move_data_from_api_to_database.py, then updated get_comment_text myself because response.status_code = 429 it threw an error when i was testing. Looks like that's resolved.

One of two of the tests are failing, can you check that out? Once resolved, I think this is good

[GregoryHo88]

FYI I merged the updated main into the branch to capture the add_comments_to_date_range function in move_data_from_api_to_database.py, then updated get_comment_text myself because response.status_code = 429 it threw an error when i was testing. Looks like that's resolved.

One of two of the tests are failing, can you check that out? Once resolved, I think this is good

roger that