Closed konklone closed 11 years ago
Here are some notes I just took from Sen. Lamar Alexander's form. The field names are gobbledygook slugs, but they seem to stay constant (even when I opened a Chrome Incognito window and visited it again). Same for the hidden Form_id
value.
<form method="post" class="uniForm">
<input type="hidden" name="IsPosted" value="true" />
<input type="hidden" name="Form_id" value="5a20ff52-c723-4c84-b2b6-8b7c009b9cf2" />
<!-- fields inside here -->
<input type="submit" value="Submit" class="primaryAction" />
</form>
Optional fields -
Prefix - field_74cbc1c2-c443-490a-b444-6106d1bf996b
Address 2 - field_5931f2b2-5b88-46b0-a7cf-5be3fd07974b
Phone - field_ef1f1e71-8ece-4f2b-a2d9-c790e0665be3
Work Phone - field_d75596f4-bd81-4fc9-81fa-7988ef9e90b2
Newsletter -
name: Subscription_id_5a20ff52-c723-4c84-b2b6-8b7c009b9cf2
value: 77e958c1-95c2-4f75-afaf-2cc45eee8682
Required fields -
First Name - field_0d9fdd23-1eb7-46e0-a061-05457b67ca76
Last Name - field_7fbed671-41c6-4fee-a8ff-88f3bc983a2d
Address 1 - field_99425453-a256-4c53-9e4d-6c3bb37d4108
City - field_a8dc4be1-b6d0-45b2-8e8c-fce92a58faa3
State - field_33ed7683-4ea7-4ecb-a7bf-af5345498ab9
Zip - field_420f4180-d327-4c63-aac5-efd047b1b463
Email - field_dfd4aea8-8a57-43d6-a4dc-f2a2971d44b3
Topic - field_50b8687d-ef75-4803-aa8f-d3c9ca3dc003
Body - field_8f2ab492-6dcd-4dd7-a35f-56574f6f6531
Topics -
<option value=""></option>
<option value="Abortion">Abortion</option>
<option value="Agriculture">Agriculture</option>
<option value="Animal_Rights">Animal Rights</option>
<option value="Banking">Banking and Finance</option>
<option value="Budget">Budget and Spending</option>
<option value="Casework">Casework</option>
<option value="Civil_Rights">Civil Rights</option>
<option value="Defense">Defense</option>
<option value="Economy">Economy</option>
<option value="Education">Education</option>
<option value="Energy">Energy</option>
<option value="Environment">Environment</option>
<option value="Foreign_Affairs">Foreign Affairs</option>
<option value="Guns_Firearms">Guns and Firearms</option>
<option value="Health_Care">Health Care</option>
<option value="Homeland_Security">Homeland Security</option>
<option value="Immigration">Immigration</option>
<option value="Information_Technology">Internet / Radio / TV</option>
<option value="Labor">Labor and Pensions</option>
<option value="National_Parks">National Parks</option>
<option value="Postal_Service">Postal Service</option>
<option value="Small_Business">Small Business</option>
<option value="Social_Security">Social Security</option>
<option value="Taxes">Taxes</option>
<option value="Trade">Trade</option>
<option value="Transportation">Transportation</option>
<option value="Veterans">Veterans</option>
<option value="Welfare">Welfare</option>
<option value="Special_Requests">Eagle Scout / Gold Award,Award, Birthday, and Retirement</option>
States -
<option value="----Military----">----Military----</option>
<option value="AA">AA</option>
<option value="AE">AE</option>
<option value="AP">AP</option>
<option value="----States----">----States----</option>
<option value="AL">Alabama</option>
<option value="AK">Alaska</option>
<option value="AZ">Arizona</option>
<option value="AR">Arkansas</option>
<option value="CA">California</option>
<option value="CO">Colorado</option>
<option value="CT">Connecticut</option>
<option value="DE">Delaware</option>
<option value="DC">District of Columbia</option>
<option value="FL">Florida</option>
<option value="GA">Georgia</option>
<option value="HI">Hawaii</option>
<option value="ID">Idaho</option>
<option value="IL">Illinois</option>
<option value="IN">Indiana</option>
<option value="IA">Iowa</option>
<option value="KS">Kansas</option>
<option value="KY">Kentucky</option>
<option value="LA">Louisiana</option>
<option value="ME">Maine</option>
<option value="MD">Maryland</option>
<option value="MA">Massachusetts</option>
<option value="MI">Michigan</option>
<option value="MN">Minnesota</option>
<option value="MS">Mississippi</option>
<option value="MO">Missouri</option>
<option value="MT">Montana</option>
<option value="NE">Nebraska</option>
<option value="NV">Nevada</option>
<option value="NH">New Hampshire</option>
<option value="NJ">New Jersey</option>
<option value="NM">New Mexico</option>
<option value="NY">New York</option>
<option value="NC">North Carolina</option>
<option value="ND">North Dakota</option>
<option value="OH">Ohio</option>
<option value="OK">Oklahoma</option>
<option value="OR">Oregon</option>
<option value="PA">Pennsylvania</option>
<option value="RI">Rhode Island</option>
<option value="SC">South Carolina</option>
<option value="SD">South Dakota</option>
<option value="Tennessee" selected="selected">Tennessee</option>
<option value="TX">Texas</option>
<option value="UT">Utah</option>
<option value="VT">Vermont</option>
<option value="VA">Virginia</option>
<option value="WA">Washington</option>
<option value="WV">West Virginia</option>
<option value="WI">Wisconsin</option>
<option value="WY">Wyoming</option>
And here are my notes for Kelly Ayotte's form:
<form id='standard_form' onSubmit='return validateStandardForm(this)' method='post' action='/?p=submit'>
<!-- fields in here -->
<input class='input-button' type='submit' name='submit' value='Submit Information' />
<input id='validate' type='hidden' name='validate' value='contact' />
</form>
Optional fields -
Suffix - suffix (free text, maxlength 30)
Street Address - address2 (free text, maxlength 50)
Home Phone - homephone (free text, maxlength 20)
Work Phone - workphone (free text, maxlength 20)
Required fields -
Prefix - prefix (free text, maxlength 30)
First name - firstname (free text, maxlength 30)
Last name - lastname (free text, maxlength 30)
Street Address - address1 (free text, maxlength 50)
City - city (free text, maxlength 35)
State - state (select)
Zip Code - zip (free text, maxlength 10)
E-Mail - email (free text, maxlength 50)
Topic - topic (select)
Subject - subject (free text, no maxlength)
Body - messagebody (free text, no maxlength)
Topics -
<option value=' '>PICK A TOPIC</option>
<option value='Abortion / Family Planning'>Abortion / Family Planning</option>
<option value='Agriculture'>Agriculture</option>
<option value='Animal Welfare'>Animal Welfare</option>
<option value='Banking / Finance'>Banking / Finance</option>
<option value='Business'>Business</option>
<option value='Defense'>Defense</option>
<option value='Education'>Education</option>
<option value='Energy'>Energy</option>
<option value='Environment'>Environment</option>
<option value='Federal Budget'>Federal Budget</option>
<option value='Fish / Wildlife'>Fish / Wildlife</option>
<option value='Foreign Relations'>Foreign Relations</option>
<option value='Government Affairs'>Government Affairs</option>
<option value='Health Care'>Health Care</option>
<option value='Housing'>Housing</option>
<option value='Immigration / Border Issues'>Immigration / Border Issues</option>
<option value='Jobs'>Jobs</option>
<option value='Judiciary'>Judiciary</option>
<option value='Justice'>Justice</option>
<option value='Labor'>Labor</option>
<option value='Medicare / Medicaid'>Medicare / Medicaid</option>
<option value='Native Americans'>Native Americans</option>
<option value='Other - Not Listed'>Other - Not Listed</option>
<option value='Pensions / Retirement'>Pensions / Retirement</option>
<option value='Postal Service'>Postal Service</option>
<option value='Science / Space'>Science / Space</option>
<option value='Second Amendment'>Second Amendment</option>
<option value='Social Security'>Social Security</option>
<option value='Taxes / IRS'>Taxes / IRS</option>
<option value='Telecommunications'>Telecommunications</option>
<option value='Terrorism'>Terrorism</option>
<option value='Trade'>Trade</option>
<option value='Transportation'>Transportation</option>
<option value='Veterans Affairs'>Veterans Affairs</option>
<option value='Welfare'>Welfare</option>
<option value='Womens Issues'>Womens Issues</option>
States -
<option value='AL'>Alabama</option>
<option value='AK'>Alaska</option>
<option value='AS'>American Samoa</option>
<option value='AZ'>Arizona</option>
<option value='AR'>Arkansas</option>
<option value='AA'>Armed Forces the Americas (AA)</option>
<option value='AE'>Armed Forces Europe (AE)</option>
<option value='AP'>Armed Forces Pacific (AP)</option>
<option value='CA'>California</option>
<option value='CO'>Colorado</option>
<option value='CT'>Connecticut</option>
<option value='DE'>Delaware</option>
<option value='DC'>District of Columbia</option>
<option value='FL'>Florida</option>
<option value='GA'>Georgia</option>
<option value='GU'>Guam</option>
<option value='HI'>Hawaii</option>
<option value='ID'>Idaho</option>
<option value='IL'>Illinois</option>
<option value='IN'>Indiana</option>
<option value='IA'>Iowa</option>
<option value='KS'>Kansas</option>
<option value='KY'>Kentucky</option>
<option value='LA'>Louisiana</option>
<option value='ME'>Maine</option>
<option value='MD'>Maryland</option>
<option value='MA'>Massachusetts</option>
<option value='MI'>Michigan</option>
<option value='MN'>Minnesota</option>
<option value='MS'>Mississippi</option>
<option value='MO'>Missouri</option>
<option value='MT'>Montana</option>
<option value='NE'>Nebraska</option>
<option value='NV'>Nevada</option>
<option selected='selected' value='NH'>New Hampshire</option>
<option value='NJ'>New Jersey</option>
<option value='NM'>New Mexico</option>
<option value='NY'>New York</option>
<option value='NC'>North Carolina</option>
<option value='ND'>North Dakota</option>
<option value='OH'>Ohio</option>
<option value='OK'>Oklahoma</option>
<option value='OR'>Oregon</option>
<option value='PA'>Pennsylvania</option>
<option value='PR'>Puerto Rico</option>
<option value='RI'>Rhode Island</option>
<option value='SC'>South Carolina</option>
<option value='SD'>South Dakota</option>
<option value='TN'>Tennessee</option>
<option value='TX'>Texas</option>
<option value='UT'>Utah</option>
<option value='VT'>Vermont</option>
<option value='VI'>Virgin Islands</option>
<option value='VA'>Virginia</option>
<option value='WA'>Washington</option>
<option value='WV'>West Virginia</option>
<option value='WI'>Wisconsin</option>
<option value='WY'>Wyoming</option>
And even though we'd be bypassing it, here's her related validation JS code, as found here:
function
validateStandardForm() {
var Y = YAHOO.util;
var els = Y.Dom.getElementsByClassName('required');
var ok = true;
var msg = "Please correct the fields marked in red.";
var radioMsg = "";
for( var i = 0; i < els.length; i++) {
// text box or textarea
if ( els[i].className.indexOf("input-text") > -1 || els[i].className.indexOf("input-textarea") > -1 ) {
if ( els[i].value.replace(/ /g, "") == "" ) {
Y.Dom.addClass(els[i], "missing");
ok = false;
} else {
// check for proper email
if ( els[i].className.indexOf("email") > -1 && !emailCheck(els[i].value) ) {
Y.Dom.addClass(els[i], "missing");
ok = false;
} else {
Y.Dom.removeClass(els[i], "missing");
}
}
}
// select box
if ( els[i].className.indexOf("input-select") > -1 ) {
var s = els[i];
if ( s[s.selectedIndex].value.replace(/ /g, "") == "" ) {
Y.Dom.addClass(els[i], "missing");
ok = false;
} else {
Y.Dom.removeClass(els[i], "missing");
}
}
if ( els[i].className.indexOf("input-radio") > -1 || els[i].className.indexOf("input-checkbox") > -1 ) {
var radio = Y.Dom.getElementsByClassName('siblings-' + els[i].name);
var atLeastOne = false;
for( var j = 0; j < radio.length; j++) {
if ( radio[j].checked ) {
atLeastOne = true;
break;
}
radioMsg = "Please choose an option from:\n\n" + radio[j].title;
}
if ( !atLeastOne ) {
ok = false;
} else {
radioMsg = "";
}
}
if (radioMsg != "") {
break;
}
}
// special validations
var validate = document.getElementById("validate");
if ( validate ) {
}
if (!ok) {
if ( radioMsg != "")
alert(radioMsg);
else
alert(msg);
return false;
}
return true;
}
function emailCheck(str) {
var at="@";
var dot=".";
var lat=str.indexOf(at);
var lstr=str.length;
var ldot=str.indexOf(dot);
if (str.indexOf(at)==-1){
return false
}
if (str.indexOf(at)==-1 || str.indexOf(at)==0 || str.indexOf(at)==lstr){
return false
}
if (str.indexOf(dot)==-1 || str.indexOf(dot)==0 || str.indexOf(dot)==lstr){
}
if (str.indexOf(at,(lat+1))!=-1){
return false
}
if (str.substring(lat-1,lat)==dot || str.substring(lat+1,lat+2)==dot){
return false
}
if (str.indexOf(dot,(lat+2))==-1){
return false
}
if (str.indexOf(" ")!=-1){
return false
}
return true
}
Oh, and /cc @angai369 @mfb @tauberer, so you guys know I'm compiling this stuff informally here.
I put together a go at a sample schema in YAML for Roy Blunt. A couple of things:
select
s such as what I've calledADDRESS_STATE_POSTAL_ABBREV
here could list their enum values there as well.contact_form
is a bit weird with the selector at the top level and the url defined in a step,
but there's not necessarily anything to do with the form's selector so I'm unsure what a better approach might look
like.Don't know what responding to a CAPTCHA challenge should look like.
name:
first: Roy
last: Blunt
id:
bioguide: B000575
govtrack: 400034
contact_form:
selector: "form.uniForm"
steps:
- visit: http://www.blunt.senate.gov/public/index.cfm/contact-form?p=contact-roy
- fill_in:
- name: field_4f83841c-3472-434f-9981-a5c91d68b253
selector: "#field_4f83841c-3472-434f-9981-a5c91d68b253"
value: NAME_PREFIX
- name: field_d9d438fc-cf81-44b8-915b-5bff36c9c025
selector: "#field_d9d438fc-cf81-44b8-915b-5bff36c9c025"
value: NAME_FIRST
required: true
- name: field_83e1f87a-3924-48b3-ae00-115adc633ae9
selector: "#field_83e1f87a-3924-48b3-ae00-115adc633ae9"
value: NAME_LAST
required: true
- name: field_51d627cb-d6ca-44c8-bc60-6ee5ebb42bc8
selector: "#field_51d627cb-d6ca-44c8-bc60-6ee5ebb42bc8"
value: ADDRESS_STREET
required: true
- name: field_2182c802-8c04-470b-965a-2b3524fd44fa
selector: "#field_2182c802-8c04-470b-965a-2b3524fd44fa"
value: ADDRESS_STREET_2
- name: field_46305235-87d9-4708-a3d5-8fc54a49fe78
selector: "#field_46305235-87d9-4708-a3d5-8fc54a49fe78"
value: ADDRESS_CITY
required: true
- name: field_383cf00f-8254-4d45-bc7e-d3c3b3a2f6b8
selector: "#field_383cf00f-8254-4d45-bc7e-d3c3b3a2f6b8"
value: ADDRESS_ZIP5
required: true
- name: field_6f798c07-d038-46ce-be19-8d706a3e6b88
selector: "#field_6f798c07-d038-46ce-be19-8d706a3e6b88"
value: PHONE
- name: field_8c4a91ac-459d-41dd-b577-43769d4f5798
selector: "#field_8c4a91ac-459d-41dd-b577-43769d4f5798"
value: EMAIL
required: true
- name: field_1fcd0a32-c2b6-40e7-9d3f-18de98bf7997
selector: "#field_1fcd0a32-c2b6-40e7-9d3f-18de98bf7997"
value: MESSAGE
required: true
- select:
- name: field_30614977-cee4-4a77-ba62-10169d06445b
selector: "#field_30614977-cee4-4a77-ba62-10169d06445b"
value: ADDRESS_STATE_POSTAL_ABBREV
required: true
options: US_STATES
- name: field_ff576494-50ce-42fc-aff0-a14f510dc4d8
selector: "#field_ff576494-50ce-42fc-aff0-a14f510dc4d8"
value: TOPIC
required: true
options:
- Abortion
- Agriculture
- Animal Rights
- Budget/Spending
- Business
- Civil Rights
- Commerce
- Congratulations/Birthdays
- Crime/Drugs
- Debt Ceiling
- Defense
- Education
- Elections
- Energy
- Environment
- Family Values
- Foreign Relations
- Health
- Homeland Security
- Housing
- Immigration
- Intelligence
- Jobs
- Judiciary
- Labor
- Medicare
- Parks and Public Lands
- Pension/Retirement
- Postal
- Regulatory Reform
- Religion
- Science/Technology
- Second Amendment
- Senate Procedure
- Small Business
- Social Security
- Taxes
- Telecommunications
- Tobacco
- Trade
- Transportation
- Unemployment
- Veterans
- Water/Rivers
- Welfare
- Other
- click_on:
value: Submit
selector: "input[type='submit']"
success:
headers:
status: 200
body:
contains: Thank you
# matches: "thank you, [a-z\. ]+ for contacting"
@drinks, I like this a lot.
For everyone else - this comes directly from Formageddon's model for submission. Important to note is that this approach would be using a headless browser to operate the form like a human would -- not to simulate a form submission by POSTing to the right URL with the right params. This is probably more realistic, but does mean maintaining things like selectors (not just the name
of the field) and submit buttons.
Also, it's entirely possible that a JS-driven form might not have all of the fields and buttons inside the <form>
tag being submitted - so I don't know if the <form>
selector is needed at all.
As for metadata about the member, I don't think anything more than a bioguide
is needed here - the rest can be gotten from connecting this dataset to the stuff in unitedstates/congress-legislators
.
Important to note is that this approach would be using a headless browser to operate the form like a human would -- not to simulate a form submission by POSTing to the right URL with the right params.
This reminds me that the form action
and method
should probably be included under contact_form
. Of course, POSTing to the designated url will most likely fail due to CSRF protection, but it probably makes sense to include them.
This is probably more realistic, but does mean maintaining things like selectors (not just the name of the field) and submit buttons.
Giving a selector here helps in cases like a set of checkboxes or radio buttons that all have the same name
attribute, so you can determine which one to check. Maybe less useful for text inputs, though, where a viable selector can be divined from the name by itself. I could go either way, but a single selector field across the board seemed simple to me.
I don't know if the
<form>
selector is needed at all
The reason I included this is to help with fields that aren't unique on a page, such as the submit button in the above example. It's just a <input type="submit" value="Submit">
, which could just as easily submit a search box. If the form selector is given, then say, a capybara script could by default run the fill_out
and click_on
steps in a within
block; likewise a jquery script could chain find
s off of that initial selector, etc.
The reason I included this is to help with fields that aren't unique on a page, such as the submit button in the above example. It's just a
<input type="submit" value="Submit">
, which could just as easily submit a search box.
That's what I figured, but I think this is better done by repeating the form selector as part of the selector for each element (e.g. form.uniForm input[type=text].name
), since the elements don't strictly need to be contained inside the <form>
element at all (if the form is submitted through JavaScript). The only time this will feel like a chore is when the <form>
parent selector is even needed, for elements without IDs or other distinguishing characteristics.
This reminds me that the form action and method should probably be included under contact_form.
But Capybara (or a similar headless-browser approach) wouldn't use these at all. I guess we could include them for others' sake, but it's not a priority to collect it for me.
Good point re: selectors, works for me.
Ok, a couple of updates:
id:
bioguide: B000575
contact_form:
method: POST
action: ''
steps:
- visit: http://www.blunt.senate.gov/public/index.cfm/contact-form?p=contact-roy
- fill_in:
- name: field_4f83841c-3472-434f-9981-a5c91d68b253
selector: "#field_4f83841c-3472-434f-9981-a5c91d68b253"
value: NAME_PREFIX
- name: field_d9d438fc-cf81-44b8-915b-5bff36c9c025
selector: "#field_d9d438fc-cf81-44b8-915b-5bff36c9c025"
value: NAME_FIRST
required: true
- name: field_83e1f87a-3924-48b3-ae00-115adc633ae9
selector: "#field_83e1f87a-3924-48b3-ae00-115adc633ae9"
value: NAME_LAST
required: true
- name: field_51d627cb-d6ca-44c8-bc60-6ee5ebb42bc8
selector: "#field_51d627cb-d6ca-44c8-bc60-6ee5ebb42bc8"
value: ADDRESS_STREET
required: true
- name: field_2182c802-8c04-470b-965a-2b3524fd44fa
selector: "#field_2182c802-8c04-470b-965a-2b3524fd44fa"
value: ADDRESS_STREET_2
- name: field_46305235-87d9-4708-a3d5-8fc54a49fe78
selector: "#field_46305235-87d9-4708-a3d5-8fc54a49fe78"
value: ADDRESS_CITY
required: true
- name: field_383cf00f-8254-4d45-bc7e-d3c3b3a2f6b8
selector: "#field_383cf00f-8254-4d45-bc7e-d3c3b3a2f6b8"
value: ADDRESS_ZIP5
required: true
- name: field_6f798c07-d038-46ce-be19-8d706a3e6b88
selector: "#field_6f798c07-d038-46ce-be19-8d706a3e6b88"
value: PHONE
- name: field_8c4a91ac-459d-41dd-b577-43769d4f5798
selector: "#field_8c4a91ac-459d-41dd-b577-43769d4f5798"
value: EMAIL
required: true
- name: field_1fcd0a32-c2b6-40e7-9d3f-18de98bf7997
selector: "#field_1fcd0a32-c2b6-40e7-9d3f-18de98bf7997"
value: MESSAGE
required: true
- select:
- name: field_30614977-cee4-4a77-ba62-10169d06445b
selector: "#field_30614977-cee4-4a77-ba62-10169d06445b"
value: ADDRESS_STATE_POSTAL_ABBREV
required: true
options: US_STATES
- name: field_ff576494-50ce-42fc-aff0-a14f510dc4d8
selector: "#field_ff576494-50ce-42fc-aff0-a14f510dc4d8"
value: TOPIC
required: true
options:
- Abortion
- Agriculture
- Animal Rights
- Budget/Spending
- Business
- Civil Rights
- Commerce
- Congratulations/Birthdays
- Crime/Drugs
- Debt Ceiling
- Defense
- Education
- Elections
- Energy
- Environment
- Family Values
- Foreign Relations
- Health
- Homeland Security
- Housing
- Immigration
- Intelligence
- Jobs
- Judiciary
- Labor
- Medicare
- Parks and Public Lands
- Pension/Retirement
- Postal
- Regulatory Reform
- Religion
- Science/Technology
- Second Amendment
- Senate Procedure
- Small Business
- Social Security
- Taxes
- Telecommunications
- Tobacco
- Trade
- Transportation
- Unemployment
- Veterans
- Water/Rivers
- Welfare
- Other
- click_on:
value: Submit
selector: "form.uniForm input[type='submit']"
success:
headers:
status: 200
body:
contains: Thank you
# matches: "thank you, [a-z\. ]+ for contacting"
Nice. Two things - small one is the bioguide
field can float to the top, since there are no other id
fields. The other is that I'm wondering how we reconcile the contact form URLs in this repository with the URLs in unitedstates/congress-legislators
. I think I saw that some members will require multiple pages to be visited in order to submit the final product?
Either way, it feels like the contact_form
field in unitedstates/congress-legislators
and the URLs to get visit
'd in this one may have to be maintained entirely separately from each other. Maybe the most efficient thing is to use contact_form
values from congress-legislators
as starting points for making new or updated descriptions of contact form submission.
I think this has stabilized enough to close this issue.
Figure out a schema for describing individual Senators' forms that would provide an automated tool enough information to submit that form.