search

openaq / openaq-upload

Batch uploader for OpenAQ
MIT License
2 stars 0 forks source link

E-mail handling #12

Closed nbumbarger closed 7 years ago

nbumbarger commented 7 years ago

The wireframes suggest that the user can expect an e-mail when processing is complete. How will this be implemented on the server, and how should the form be designed to accommodate this feature? For example:

jflasher commented 7 years ago

The trick here is that none of the adapters (where the data is brought in to be ingested) touch the database and it'd be nice to keep it that way. I think the easiest way to handle this within openaq-fetch is if you just ask them at submit time what their email address is if they want to be emailed (either modal or just one extra form input field) and just stick that on every measurement. It'll be somewhat needlessly ugly, but they'll never see the final submitted CSV. This way, the adapter will have the email at submit time. I had thought we'd do a separate meta file for each measurements CSV that'd have this info, but not sure we need that if this is the only thing that'd go in it.

nbumbarger commented 7 years ago

Alright, that seems like a fine workaround.

nbumbarger commented 7 years ago

I thought of a potential implementation problem, which may not actually be a problem, but the S3 bucket is expecting a file, correct? Right now, the page parses the input as JSON then sends the original file to the pre-signed URL- it doesn't need to actually modify the CSV data. Would it be possible to modify the CSV data inline then send that data to the bucket, rather than the file object?

nbumbarger commented 7 years ago

@jflasher I haven't been able to find much about uploading non-file data to S3, although it seems like it would have to be possible. Do you think that constructing a file object, eg. this post, might do the trick? Or is there a better-sanctioned way to go about it?

jflasher commented 7 years ago

When testing before, I was just uploading text where the file is currently being uploaded so I think you can upload whatever string you want there. You should have S3 credentials to download and check whatever was uploaded.

On November 22, 2016 at 21:09:39, Nick Bumbarger (notifications@github.com(mailto:notifications@github.com)) wrote:

@jflasher(https://github.com/jflasher) I haven't been able to find much about uploading non-file data to S3, although it seems like it would have to be possible. Do you think that constructing a file object, eg. this post(http://stackoverflow.com/questions/8390855/how-to-instantiate-a-file-object-in-javascript), might do the trick? Or is there a better-sanctioned way to go about it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub(https://github.com/openaq/openaq-upload/issues/12#issuecomment-262274713), or mute the thread(https://github.com/notifications/unsubscribe-auth/AAz0JkjdH3un1C1jjBa7uzF8BkBuAP3qks5rAwy7gaJpZM4K4Yj6).

nbumbarger commented 7 years ago

Hmm, I got a somewhat contradictory error when I pasted the exact test you used in this issue (using the pre-signed URL from the API) into the uploader page. It's

Upload successful!  Server responded with: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ASIAJY4C5RDHCG7DEXCA</AWSAccessKeyId><StringToSign>PUT

1479942915
x-amz-security-token:FQoDYXdzEK3//////////wEaDOVEuDVMzjgiVO7XtSKqA6xl5ddRoTWAhxKh5cYIzduZ78IEoK3+AJ/Pr1GPR4EX8HDCdp3XLR0e6ej0vXObc+L2IerrP8dk7DLEQtgnQURencmBXVWh6wTgxKeB0XIBVaSh+kdRCcqccmYJ7s2w/Iz/E4U7T4HcwSW8W9JyE4D7wb1tnRMUwv4tsqsKMZtevY0R40u/Tp54ho2R2khC5ZZDzSGYTNp4NBoqJP5rW2ajRMSdvwWGyg7AzzMczxuN5cu/tkGAY6phyfjX8FfaoOK2rE7zmGXDP62LpeCBMYy/6/ls5bHgIsR2CDj3Vu07PHN3ytVOIIOqwN+SOqgnDj1RbQaCz3bvfxAZj9FLNGFA2MBV3qXx7NFZtEr1DHCQ0dxJE2gZW5gg4mOW98QSQTXZwVntO3+47hsvp86wJ92aE/QvC6fTUpKugw4lesgQ7E0KE2yTykfTbAku3+91FMCACJbvipPXWDAEICk24DSiZ9ZjgwOyoZRq0aDX9CWfWmq3/ZFR4J6sd5rzGqC+kHLtlVC4p9LcpBtY2zLxKR9tlwiXtlM4Q6XdyozwH8ycjVKGeZLX22+0vSiP7dfBBQ==
/openaq-uploads-staging/passing-with-optional-columns.csv</StringToSign><SignatureProvided>0cpDMztYkww8TOwucCNxzeqJOEI=</SignatureProvided>

Any ideas? I'm out tomorrow but I will be back at work on Friday. Happy Thanksgiving!

jflasher commented 7 years ago

You'd need to check headers to be sure, but I bet it's because we're now including content-type in signed string and this doesn't pass it.

On November 24, 2016 at 04:32:48, Nick Bumbarger (notifications@github.com(mailto:notifications@github.com)) wrote:

Hmm, I got a somewhat contradictory error when I pasted the exact test you used in this(https://github.com/openaq/openaq-upload/issues/10#issue-189469267) issue (using the pre-signed URL from the API) into the uploader page. It's

Upload successful! Server responded with: <?xml version="1.0" encoding="UTF-8"?> SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.ASIAJY4C5RDHCG7DEXCAPUT 1479942915 x-amz-security-token:FQoDYXdzEK3//////////wEaDOVEuDVMzjgiVO7XtSKqA6xl5ddRoTWAhxKh5cYIzduZ78IEoK3+AJ/Pr1GPR4EX8HDCdp3XLR0e6ej0vXObc+L2IerrP8dk7DLEQtgnQURencmBXVWh6wTgxKeB0XIBVaSh+kdRCcqccmYJ7s2w/Iz/E4U7T4HcwSW8W9JyE4D7wb1tnRMUwv4tsqsKMZtevY0R40u/Tp54ho2R2khC5ZZDzSGYTNp4NBoqJP5rW2ajRMSdvwWGyg7AzzMczxuN5cu/tkGAY6phyfjX8FfaoOK2rE7zmGXDP62LpeCBMYy/6/ls5bHgIsR2CDj3Vu07PHN3ytVOIIOqwN+SOqgnDj1RbQaCz3bvfxAZj9FLNGFA2MBV3qXx7NFZtEr1DHCQ0dxJE2gZW5gg4mOW98QSQTXZwVntO3+47hsvp86wJ92aE/QvC6fTUpKugw4lesgQ7E0KE2yTykfTbAku3+91FMCACJbvipPXWDAEICk24DSiZ9ZjgwOyoZRq0aDX9CWfWmq3/ZFR4J6sd5rzGqC+kHLtlVC4p9LcpBtY2zLxKR9tlwiXtlM4Q6XdyozwH8ycjVKGeZLX22+0vSiP7dfBBQ== /openaq-uploads-staging/passing-with-optional-columns.csv0cpDMztYkww8TOwucCNxzeqJOEI=

Any ideas? I'm out tomorrow but I will be back at work on Friday. Happy Thanksgiving!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub(https://github.com/openaq/openaq-upload/issues/12#issuecomment-262651177), or mute the thread(https://github.com/notifications/unsubscribe-auth/AAz0JudMQEHC8S_HhmXXI8t2J5PJPB7Iks5rBMYYgaJpZM4K4Yj6).

nbumbarger commented 7 years ago

https://github.com/openaq/openaq-upload/pull/16 adds email as an additional column in the transmitted CSV data.