search

codeforamerica / MuniciPal

:speech_balloon: Consulting city-dwellers about legislation near them.
22 stars 14 forks source link

broken rake helper task for fetching Legistar based structure #167

Closed techieshark closed 7 years ago

techieshark commented 7 years ago

To aid with creating db migrations, we have a rake task which scrapes the Legistar documentation and determines what the corresponding rails database schema types should be for a given endpoint.

This recently broke, apparently because of a change to the scraper system being used.

Here's what we get now (broken):

rake legistar_matters:structure
I, [2017-04-19T13:10:20.636123 #3877]  INFO -- : Connection opened to http://webapi.legistar.com and logging to log/fetch-legistar.log
E, [2017-04-19T13:10:21.000020 #3877] ERROR -- : Failed fetching for query_url: http://webapi.legistar.com/Help/Api/GET-v1-Client-Matters, body: { "input": { "webpage/url": "http://webapi.legistar.com/Help/Api/GET-v1-Client-Matters"}}; status: 400, error:

Here's what we expect - something like:

rake legistar_matters:structure
I, [2017-04-19T13:10:42.461492 #3999]  INFO -- : Connection opened to http://webapi.legistar.com and logging to log/fetch-legistar.log
t.integer :id
t.string :guid
t.datetime :last_modified_utc
t.string :row_version
t.string :file
t.string :name
t.string :title
t.integer :type_id
t.string :type_name
t.integer :status_id
t.string :status_name
t.integer :body_id
t.string :body_name
t.datetime :intro_date
t.datetime :agenda_date
t.datetime :passed_date
t.datetime :enactment_date
t.string :enactment_number
t.string :requester
t.string :notes
t.string :version
t.string :text1
t.string :text2
t.string :text3
t.string :text4
t.string :text5
t.datetime :date1
t.datetime :date2
t.string :ex_text1
t.string :ex_text2
t.string :ex_text3
t.string :ex_text4
t.string :ex_text5
t.string :ex_text6
t.string :ex_text7
t.string :ex_text8
t.string :ex_text9
t.string :ex_text10
t.datetime :ex_date1
t.datetime :ex_date2
t.datetime :ex_date3
t.datetime :ex_date4
t.datetime :ex_date5
t.datetime :ex_date6
t.datetime :ex_date7
t.datetime :ex_date8
t.datetime :ex_date9
t.datetime :ex_date10

So two very related issues, one being the broken scraper and the other being logging output that doesn't help us realize what the problem is caused by.