jbrzusto
commented
7 years ago Nice bug!
The problem is in the R Rook package I'm using as the server back-end; it is url-escaping the POST request body even though that has already happened upstream of it (Rook hasn't kept up with changes to the R tools package, on whose Rhttpd server it depends, I think).
One result is that if a POST query item contains a '+' character, that is being converted to a ' ' (space). And '+' is one of the characters used in standard base-64 encoding, which is how the
authToken is constructed from random bytes. In fact, Rook seems to do a bunch of crufty parsing that's no longer necessary, so I'll bypass that and grab the parent frame's postBody object which already has
form items correctly unescaped and split by field name.
The initial request to obtain the authToken seems to work consistently, but not the subsequent requests to get the list of batches:
First Request: https://sgdata.motus.org/data/custom/authenticate_user?json={"password":"xxxxxx","user":"sgdata.reader"}
First response: {"projects":{"88":"Monomoy - Steph Koch","89":"TBrown - NSWO","110":"Holberton 2015-16","111":"LPBO-Moult Migrants","112":"BSC-Common Nighthawks","113":"Wageningen University & Research","114":"Europe - Bats","115":"Long Point Vagrants","116":"Immune Function and Migration in Song Sparrows","117":"French Guiana Shorebirds","90":"Gerson - Florida Gulf Coast - Apalachicola","91":"SCraik - NSWO","118":"Tadoussac NSWO","119":"Missouri - E. Whip-poor-will","92":"Ontario Barn and Cliff Swallows","93":"Pitman - Green Darner","94":"Cliff Lemen 2015","95":"BellVagrantsNS","96":"Lefevre - Florida Gulf Coast University","97":"BSC tags for later distribution","10":"Taylr","98":"Zvenigorod","99":"Atlantic Darners","11":"Peterson","12":"Smetzer","13":"Semipalmated Sandpipers in the Bay of Fundy","14":"Loring","15":"Aubry","16":"Boyles","17":"Chapman","18":"Dossman_Ohio","19":"Dossman_Jamaica","120":"Western Gulf Avian Resource Assessment","121":"Maggini - Eastern Austria","0":"Motus Test Project","1":"Motus Ontario Array","122":"Rushton Woods","2":"Motus Atlantic Array","123":"Vogelwarte Helgoland","3":"EC-Quebec-St.Laurent","124":"NJA - South America Shorebirds","125":"Guyra Paraguay","4":"Region 4 USFWS ","5":"EC - Atlantic","126":"Louisiana Yellow Rail","127":"Knockaert Raptors","6":"Panama Gateway ","128":"Panama Prothonotary Warbler","7":"Welch","8":"Holbterton 2013-14","129":"University of Oklahoma","9":"Parker River\/Great Bay Refuge","20":"Gugli","21":"Gull2013","22":"IPSP2013","23":"Motus","24":"Norris","25":"obrien","26":"Salda - Atlantic Bank Swallows","27":"Smith","28":"IPSP2016","29":"R5_SALS","130":"Chiloe Hudsonian Godwit","131":"York University Killdeer","132":"Least Tern - Microsite Assessment of Range Ecology","133":"LINWRC Bird and Bat Tracking","134":"Autumn Migration ofNorthern Myotis in PA","135":"Maggini - Ventotene spring","137":"Abiotic and biotic influences on Snowfinch","138":"Montreal tag sensitivity","139":"Barbados","30":"Range Tests","31":"Coastal Louisiana Array","32":"Dowling","33":"Ernst","34":"Gratto","35":"Powdermill Nature Reserve","36":"Chickadee natal dispersal and social behavior","37":"Wheeler - Loggerhead Shrike","38":"James Bay Shorebirds","39":"Nol\/Parada- Blackpoll Warbler Migration","140":"SCDNR Shorebirds","141":"BDs_201704","142":"Effects of mercury on ovenbird migration","143":"Clapper Rail Survival in CT","145":"Kirtland's Warbler Tracking, Bahamas - Michigan","146":"Neonicotinoid insecticides and migratory birds","147":"Southern Great Plains Snowy Plover","148":"Indiana","149":"Belize","40":"EC - Arctic Shorebirds","41":"Max Planck ","42":"Tremblay - Boreal Woodpeckers as forest indicators","43":"Eastern Shore of Virginia National Wildlife Refuge","45":"BI_MERL","46":"Stutchbury - Ontario Wood Thrush","47":"Red Knot staging and migration ecology.","48":"Northeast Motus Collaboration","49":"Thrush post-breeding ecology in Quebec","150":"Coastal Georgia Array","151":"Fort Indiantown Gap Bats","152":"New York Lake Plains Motus","153":"Stopover Ecology of Semipalmated Sandpipers NGOM","154":"Ontario Pigeons","155":"Johnson - SE Florida Coast","156":"Black Tern movements - breeding\/migration","157":"Bioenergetics of WNS in western bat species","158":"Eastern Ontario Grasshopper Sparrow","159":"Local and migration movements of Canada Warbler","50":"Morbey - Protandry","51":"Nol - Wintering Shorebirds","54":"Indiana Bats - 2015","55":"Norris - MOBU 2015","56":"Pennsylvania Audubon - Presque Isle Migrants","57":"Selva Colombia","58":"Semipalmated Sandpiper stopover in Plymouth, MA","59":"NJ Audubon - Delaware Bay Shorebirds","160":"Myotis movements in eastern Québec","161":"Migration connectivity of the Rusty Blackbird ","162":"Western MA Tree Swallows","163":"Norris - MOBU Neonic 2017","164":"Tracking post-breeding movements of Purple Martins","165":"Southern Michigan Motus Array","166":"Ft. De Soto","167":"Virginia Eastern Red Bats","168":"Montreal moult migrant research","169":"Species at Risk Research on the Carden Alvar","60":"Mitchell - Canada Warbler - Stopover Ecology","61":"EC Auks","62":"Western Chickadees","63":"Saskatchewan Migratory Shorebirds","64":"Ohio State University","65":"Texas Gulf Coast Migratory Shorebirds","66":"New project (#66)- EBM_Coats","67":"Nol - Churchill Shorebirds","68":"Arctic Shorebirds - CWS Yellowknife","69":"Hilliardton Marsh Research and Education Centre","170":"Comparative physiology of Migratory songbirds","70":"Ontario Bank Swallows","71":"Lake Erie Bats ","72":"Atlantic Canada BITH","73":"Mississippi Clapper Rail","74":"CT River Valley Migratory Songbird Study","75":"AlaskaBats","76":"Control of nocturnal departure times","77":"NJ Chick Survival","78":"Atlantic Canada Shorebirds","79":"Kent State - Ohio Bats ","100":"Stutchbury - Ontario Savannah Sparrow","101":"Sandiland 2016: Ontario Myotis","102":"SentinelTag","103":"NYC Audubon","104":"Perlut - Maine Forest Birds","105":"EC - Phalaropes","106":"Gros Morne National Park","107":"Acadia National Park","80":"Shoener Illinois Bats","81":"British Columbia - Lower mainland Barn Swallows","108":"SFU - Paracas Shorebirds","109":"BerriganSWTH2016","82":"Maine - American Woodcock","83":"BIMYWA","84":"University of Connecticut - Saltmarsh Sparrow ","85":"Canadian Snow Bunting Network","86":"Morgan Brown - Blackpoll Warbler Migration Ecology","87":"Post-breeding Movements of BLPW and MYWA"},"receivers":{},"authToken":"VhMT1nQNeZJghXc+b\/096lhMvgAD3bxvrysZHX2weYjn","expiry":1.5042963027528E9,"userID":2643}
Second request, with authToken obtained above. This one fails most of the time, but appears to work at times, with no predictable pattern.
https://sgdata.motus.org/data/custom/batches_for_receiver?json={"authToken":"VhMT1nQNeZJghXc+b\/096lhMvgAD3bxvrysZHX2weYjn","batchID":0,"deviceID":522}
Most of the responses are simply:
{"error":"authorization failed"}
Some of the time, I get a valid response for the same request (with a different or the same authToken, there doesn't seem to be any consistency about re-using a token that has worked vs. another one that has failed).
E.g.
{"numHits":[3237,3237,3237,3237,3237,3237],"tsStart":[1.3967183832335E9,1.3967183832335E9,1.3967183832335E9,1.3967183832335E9,1.3967183832335E9,1.3967183832335E9],"motusDeviceID":[542,542,542,542,542,542],"motusUserID":["NA","NA","NA","NA","NA","NA"],"motusProjectID":["NA","NA","NA","NA","NA","NA"],"tsEnd":[1.50183328703E9,1.50183328703E9,1.50183328703E9,1.50183328703E9,1.50183328703E9,1.50183328703E9],"batchID":[99,99,99,99,99,99],"motusJobID":[514,514,514,514,514,514],"monoBN":[1,1,1,1,1,1],"ts":[1.5039339258849E9,1.5039339258849E9,1.5039339258849E9,1.5039339258849E9,1.5039339258849E9,1.5039339258849E9]}