search

uchicago-capp-30320 / CivicLens

Putting the public back in public commenting
https://civic-lens.org
GNU Affero General Public License v3.0
2 stars 1 forks source link

shell script to run nlp #160 #161

Closed abejburton closed 6 months ago

abejburton commented 6 months ago

Shell script in the same format as Reza's current scripts that runs the nlp pipeline and logs the success or failure time.

abejburton commented 6 months ago

cc @jgibson517 . FYI the code to run these scripts via cron job is not in github because it has reza's full server path

jgibson517 commented 6 months ago

@abe I'm getting an error on the SQL statement in the pipeline

if last_updated is not None: docs_to_update = f"""SELECT document_id FROM regulations_comment rc1 WHERE posted_date >= TIMESTAMP '{last_updated}' GROUP BY document_id HAVING COUNT() > 20 AND COUNT() >= 0.1 ( SELECT COUNT() FROM regulations_comment rc2 WHERE rc2.document_id = rc1.document_id GROUP BY document_id HAVING COUNT(*) > 20;

here's the error: RuntimeError: Error while connecting to PostgreSQL: syntax error at or near ";"

abejburton commented 6 months ago

reviewed by reza and jack