HTTP Post request to the City of Pittsburgh

Zaxcoding commented 10 years ago

Synopsis

I'm going to start working on the actual process of sending our users' information to the City of Pittsburgh.

Details:

I'm going to start by analyzing the City of Pittsbugh's site and making some notes of how they do what they do.
I'm going to send an email to the city asking if they have any way they'd prefer us to send our issues to them (perhaps a different interface than they use for the website). I know it's unlikely they actually reply, but at least trying would probably look really good.
Once it's clear where the data is going when you press 'submit' on the website and how it's formatted, I'll write a clear explanation of that.
With the design in place, we can actually implement it for our app.
Discussion
Thoughts, questions, concerns? Talk below.

Zaxcoding commented 10 years ago

Some notes: The default latitude and longitude they use for their map is 40.436757,-79.970856

On clicking 'submit' on the first page, the site makes a post request to http://www.pittsburghpa.gov/eng/process/311new.htm with the headers:

fname:
lname:
address:
city:
state:
zip:
email:
email2:
dphone:
lphone:
comment:
latbox:
lonbox:
complainant_address:
uip:
hpot:

The uip is used to mark spam, as manually visiting the page returns some php errors and marks your ip as spam.

The response to the POST is a redirect to the image page, embedded with your session id (ex: http://www.pittsburghpa.gov/311/form.htm?m=img&s=82942)

When you choose to upload a picture, you make the following post request:

POST /311/uploader.htm?m=new&id=82942 HTTP/1.1
Host: www.pittsburghpa.gov
Connection: keep-alive
Content-Length: 45189
Cache-Control: no-cache
Pragma: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://www.pittsburghpa.gov
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarymvyOHXEXUeLyAek9
Referer: http://www.pittsburghpa.gov/311/form.htm?m=img&s=82942
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: PHPSESSID=sbf00oj1mub7bma47sh6hu7or7; __utma=149580329.146670521.1395785489.1395785489.1395785489.1; __utmb=149580329.6.10.1395785489; __utmc=149580329; __utmz=149580329.1395785489.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)

It seems like the most important parts are that it expects a gzipped jpg < 5 mb in size. Once you make this request, your response (assuming you have a valid ticket number, you set the img flag to new, and you uploaded a good image) takes you to a page where you can choose a different image to upload or to submit.

Finally when you hit 'looks good' it seems like it just takes you back to the main page. You've already submitted all your information so you're really done.

Zaxcoding commented 10 years ago

Overall it looks good on the surface. A few POST requests are made with the data the user is sending, and aside from some issues like getting the PHPSESSID in our cookie, we can certainly generate all the data that they want in the POST headers.

I emailed the webmaster and told him about our project, so we'll see if there are any leads there.

I'm encouraged to attempt to send POST requests to the site (probably through a C# or Java command-line app at first), and see what kind of responses we get. If the server seems to respond well then we've got great news.

Zaxcoding commented 10 years ago

Great news. I made some POST requests via Netcat on the command line.

POST /eng/process/311new.htm HTTP/1.1
Host: www.pittsburghpa.gov
Connection: keep-alive
Content-Length: 339
Cache-Control: no-cache
Pragma: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://pittsburghpa.gov
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://pittsburghpa.gov/311/form
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: PHPSESSID=sbf00oj1mub7bma47sh6hu7or7; __utma=149580329.146670521.1395785489.1395785489.1395803899.2; __utmb=149580329.1.10.1395803899; __utmc=149580329; __utmz=149580329.1395785489.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided)

fname=Test&lname=LastName&address=HomeAddress&city=Pittsburgh&state=PA&zip=15213&email=email%40email.com&email2=&neighborhood=South+Oakland&dphone=%28123%29+456-7890&lphone=&topic=Abandoned+Vehicles&comment=I+selected+%27abandoned+vehicles%27.&latbox=40.44041496678215&lonbox=-79.96381759643555&complainant_address=&uip=67.165.106.35&hpot=

successfully returned

HTTP/1.1 302 Found
Date: Wed, 26 Mar 2014 03:41:37 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.23
Location: http://www.pittsburghpa.gov/311/form.htm?m=img&s=82950
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 107
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

which is exactly what I receive via Chrome. This is great news. It means that we can simply drop the correct info into the parameters, adjust the user agent and such if we want to be kind, and get a valid ticket id back from them (note that s=82950 is a unique ticket id generated by my request).

Zaxcoding commented 10 years ago

As far as uploading the image, I've found some resources specifically for doing that with Android (especially this look http://stackoverflow.com/questions/2935946/sending-images-using-http-post) It looks to be too difficult to test via command-line, however I successfully performed a post request to test we could get through to the image uploader.

POST /311/uploader.htm?m=new&id=82942 HTTP/1.1
Host: www.pittsburghpa.gov
Connection: keep-alive
Content-Length: 45189
Cache-Control: no-cache
Pragma: no-cache
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://www.pittsburghpa.gov
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundarymvyOHXEXUeLyAek9
Referer: http://www.pittsburghpa.gov/311/form.htm?m=img&s=82942
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Cookie: PHPSESSID=sbf00oj1mub7bma47sh6hu7or7; __utma=149580329.146670521.1395785489.1395785489.1395785489.1; __utmb=149580329.6.10.1395785489; __utmc=149580329; __utmz=149580329.1395785489.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided

m=new&id=82950

which successfully returned

HTTP/1.1 302 Found
Date: Wed, 26 Mar 2014 03:55:28 GMT
Server: Apache/2.2.14 (Ubuntu)
X-Powered-By: PHP/5.3.2-1ubuntu4.23
Location: http://www.pittsburghpa.gov/311/form.htm?m=noimg&s=82942
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 99
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

So the bottom line is great news. We absolutely have the ability to upload our information to the City of Pittsburgh website, fooling it into thinking we're using its online forms. Even without coordination from the city, we can still easily have a functional backend.

RUlanowicz commented 10 years ago

This is really awesome. That's a huge part of the final project.

RUlanowicz / Pittsburgh311

HTTP Post request to the City of Pittsburgh #4

Synopsis

Details:

Discussion