Elastic Transcoder - Githubissues

cdornan commented 11 years ago

I am expecting to build out an interface to this service fairly soon for a project, and I thought it would be a good idea to check in here to see if you are already doing this or would be interested in extending the AWS package to cover this. (I will of course be very happy to feed the code back into the AWS package.)

aristidb commented 11 years ago

I would love to have this in aws, but do not currently have plans to write the bindings myself. So, if you want to write the code and add it to aws (optimally in the form of a pull request) - please do!

cdornan commented 11 years ago

Great! -- will let you know when I have something,

cdornan commented 11 years ago

I think this one will be best tackled with a JSON package, and was proposing to use aeson (which also pulls in syb and unordered-containers). Would that be OK for you?

aristidb commented 11 years ago

Aeson is OK. Am 30.01.2013 11:29 schrieb "Chris Dornan" notifications@github.com:

I think this one will be best tackled with a JSON package, and was proposing to use aeson (which also pulls in syb and unordered-containers). Would that be OK for you?

— Reply to this email directly or view it on GitHubhttps://github.com/aristidb/aws/issues/57#issuecomment-12883390.

cdornan commented 11 years ago

After a much longer delay than I was expecting, I have an Elastic Transcode Create Job command that appears to be working. (Though some of the subsystems have been quite well tested, the command itself has not been.)

I have decided to call the new sub-system Ets. The developer docs suggested that 'aws' should be used to identify the service in the credentia scope -- that turns out to be a lie but it seemed to be a good idea anyway.
Part of the reason this has been delayed is that I got tangled up in AWS Signature v4 and decided to create a separate subsystem for this and test it with the AWS4 testsuite. I have put Sign4 under Aws.Ets for now but it could be lifted up to Aws.
The Elastic Transcode service is based on JSON requests and responses so I have modeled my requests and responses types closely on the JSON structures and used Aeson's ToJSON and FromJSON classes to map between them. I have created a testsuite that round-trips each type looking for non-identities.
I have various other observations and advice I want to ask but I thought it best to just let you have a look for now and let me know what you think.

cdornan commented 11 years ago

Sorry I left out the link to the repo I am working on above. It is here. I am not sure whether a request to pull is appropriate at this stage -- please shout if that is the way you would like to go.

aristidb commented 11 years ago

OK, cool. I'll have a closer look as soon as I have time, but for now I have this suggestion: Name the modules Aws.ElasticTranscoder.*. Ets is a bit cryptic, and "Elastic Transcoder" seems to be how they call it in the docs.

cdornan commented 11 years ago

Agreed -- the next thing I will do.

cdornan commented 11 years ago

I have renamed the modules to Aws.ElasticTranscoder and added all of the job commands. (Elastic Transcoder has three sets of commands: for managing presets, jobs and pipelines).

To get the Signature V4 to work I had to switch the date format to simplified ISO-8601 ("%Y%m%dT%H%M%SZ"); I did this by filling out a date entry in sqOtherHeaders and modifying Aws.Core to check for such a date entry and refrain from adding its own when it finds a date binding.

My branch is here.

aristidb commented 11 years ago

Would it be possible to move the v4 signature code into the aws core? This might also help avoid hacks with sqOtherHeaders. I'm currently on the road so I'm afraid I can't yet take a closer look. Am 17.02.2013 15:18 schrieb "Chris Dornan" notifications@github.com:

I have renamed the modules to Aws.ElasticTranscoder and added all of the job commands. (Elastic Transcoder has three sets of commands: for managing presets, jobs and pipelines).

To get the Signature V4 to work I had to switch the date format to simplified RFC-8601 ("%Y%m%dT%H%M%SZ"); I did this by filling out a date entry in sqOtherHeaders and modifying Aws.Core to check for such a date entry and refrain from adding its own when it finds a date binding.

My branch is here https://github.com/Lainepress/aws/tree/ets.

— Reply to this email directly or view it on GitHubhttps://github.com/aristidb/aws/issues/57#issuecomment-13686455.

cdornan commented 11 years ago

I will move it into AWS.Core (but I think it is worth putting it into its own module).

AFAICT, different signing methods have conflicting requirements on the format of the date; for example it seems as if S3 needs the current RFC-822 based format while Signature v4 needs the ISO format. It is probably best if the signature code generates the date string (it needs it to calculate the signature) in which case we need to place the date string somewhere SignedQuery. I thought placing it in sqOtherHeaders was quite neat as it avoided any change for all the other systems and only a small patch to queryToHttpRequest.

cdornan commented 11 years ago

I have moved the contents of Sign4 into Aws.Core and moved Sign4.Test to Aws.Signature4Test.

If you come round to thinking it would be better in a separate module (e.g., Aws.Core.Signature4, with non-test entry points re-exported from Aws.Core) then I will be happy to relocate.

My thinking is that these signing procedures are quite a thing onto themselves with their own test suites -- worth going into a separate module I would say.

(For convenience, my branch is here.)

cdornan commented 11 years ago

I have integrated/merged the date formatting of the Signature v4 generator into the extant Aws.Core fmtTime and added some comments.

Next up:

add the remaining operations
work out how to package the general test suite.

I am thinking the general test suite is better as a programme rather than a cabal test suite as it will in general need some prep by each user to run.

(My branch is here.)

cdornan commented 11 years ago

I have now added the Pipeline group of commands. [Correction -- I have just noticed that I stopped short but it should anyway be fairly quick to add the missing commands.]

I just need to add the preset commands and I will have the set -- I should be able to get them added later this week.

(My additions are here.)

cdornan commented 11 years ago

As noted on #63 I have completed all of the ElasticTranscoder commands now. All that remains is some testing and repackaging before merging. I expect to have completed these over the weekend.

(My branch is here.)

aristidb commented 11 years ago

Looks like you have written the only component in aws with actual unit tests.

You seem to however have some opaque binary files in your branch. You should probably get rid of them. Files like https://github.com/Lainepress/aws/blob/ets/aws4_testsuite/._get-header-key-duplicate.authz

I can't really look deeply into your changes now, but if you have cleaned this up, and want to try to get this into the next release (to be honest: not sure if that will work out), feel free to send a pull request. Also send a pull request if it doesn't have to be the next release. ;)

cdornan commented 11 years ago

For sure it will have to be ready to go if it is going to get into this release, otherwise the next one.

I still have some minor cleaning up to but hadn't realised some dot files had snuck in -- I will fix this and the other matters over the w/e and request a pull once I have everything ready.

aristidb / aws

Elastic Transcoder #57