Solution with subprocess and unoconv

datamade / django-councilmatic

:heartpulse: Django app providing core functions for *.councilmatic.org

http://councilmatic.org

MIT License

26 stars 16 forks source link

Solution with subprocess and unoconv #165

Closed reginafcompton closed 6 years ago

reginafcompton commented 6 years ago

A note about killing LibreOffice:

unoconv expects a SIGTERM (not a SIGKILL): https://github.com/dagwieers/unoconv/blob/master/unoconv#L1198

To close LibreOffice as a daemon process, you need to terminate it.

reginafcompton commented 6 years ago

Functional solution: use unoconv to convert rtf to html!

TODO:

write a script that runs the unoconv process, after import_data, i.e., we'll add the rtf to full_text, then we'll run a script that converts the rtf to html.
install unoconv and libre office on staging server (for testing) - see instructions in metro-pdf-merger: https://github.com/datamade/metro-pdf-merger#ubuntu ... if it works, then do the same for the Councilmatic server

reginafcompton commented 6 years ago

@fgregg - I made some changes to the sql and its parameters...you can can see what I've done, but in short, we needed to unpack a list of dicts and wrap the query call in sqlalchemy's text function. It did the job - let me know if you'd like to discuss on Monday.

reginafcompton commented 6 years ago

The full run on the Councilmatic server is still running: it has completed a little over 17000 bills.

It seems possible that the script could fail. So, I've added an option to run the conversion script to only update bills without html_text. Then, we'd need not start at the beginning.... @fgregg - let me know what you think.