Closed alianos- closed 9 years ago
@alianos- so we store our subjects on s3 for distribution. Sounds like you have your own subject hosting service so you don't need to leverage this but those URL's will be stored in the subject locations. It sounds like your front end uses the metadata fields to link the coords and the image, e.g.
{ "subjects": [
{
"id": "1",
"locations": [
{
"image/jpeg": "http://s3.amazonaws.com/subjects/1.png"
}
],
"metadata": {
"file_location": "http://WMS_SERVER_URL/file_1.image_ext",
"coords": [ 1, 1 ]
},
} ]
}
If this is the case then it's up to your front end to understand the custom metadata hash URL format. To avoid this error you can just edit your aws.yml config for the development and the production env (if / when you run in production mode), make it look like the test env if you want but those values will propogate to your 'locations' object (but you'll ignore them anyway).
As our API is not at version 1 yet (it's getting there). We've got work planned to use different subject storage locations via #425. This will be by configurable and you should be able to specify location URL's if you've got them hosted elsewhere.
Still what happens is inconsistent, since the server returns an error while the operation has in fact been successful. But I am assuming it will be resolved by the time you have the final version, so anyway.
Keep in mind that not all crowdsourcing projects will require external resources. For my PhD project we only had plain text multiple choice questions, so there was no need for any kind of location at all, and the same stand for this project (unless it would make sense to store plain text files with the coordinates and link to them? not sure).
Finally -having 0 understanding of what aws.yml does or how to configure it- I copied the values from the test to development and the error went away. For reference, here is what is inserted in both cases.
for reference my subjects look like that
{
"id": "3",
"metadata": {
"coordinates": "{x : 2222, y: 555}"
},
"locations": [
{
"text/plain": "https:///1/0/f119841a-3ff0-4d55-ac31-a3d14fff92b4.txt"
}
],
[...]
}
"id": "4",
"metadata": {
"coordinates": "{x : 2222, y: 555}"
},
"locations": [
{
"text/plain": "https://s3.amazonaws.com/s3_subject_bucket/test_bucket_path/1/0/6293560e-f198-4975-bfea-1d5d189326cd.txt?AWSAccessKeyId=test_key_id&Expires=1422988220&Signature=eKEXZg5I4w60OZoNd5qp9Euw%2FqA%3D&response-content-type=text%2Fplain&x-amz-acl=public-read"
}
],
[...]
PS. I am not closing the issue in case you want to add something, but feel free to close it.
So those subject locations you submitted are overwritten with the value from the AWS confg, see here. You can override the behaviour via aws.rb or just override build_resource_for(create|update) via here. Alternatively as i said before just store the location in the metadata. It'll get serialised as a response to the client and you can just ignore the locations array client side.
Also aws.yml configures the access keys and storage areas for aws s3.
We're currently building what we need as an open source project. Long term we will extract a core version of Panoptes with plugin features for this sort of thing. Feel free to add a PR to add a subject storage adapter for us ;).
I still don't get you. I did not submit any location, and I don't need any location either, so why would I store an empty location it the metadata?
here is how my request looks
'subjects': {
'locations': ['text/plain'],
'metadata': {
'coordinates': '{x:111,y:222}'
},
'links': {
'project': '1'
}
}
I would not trust myself overriding anything as I am not familiar with Ruby, and it is not the right time to learn either I am afraid :( So long as the previous solution works I am fine with it, but what I meant is that maybe you'd like to make the location optional.
As for your last sentence, I don't know what is a PR, or a 'subject storage adapter', so I am probably not the right person to contribute, at least not at this stage :(
When you submit locations via the mime type they will create s3 links where you can then put/upload your subject files too. You aren't using this obviously but those locations to s3 are being created and stored on your subjects, see the second subject you referenced here.
Re why would you store an empty location in the metadata, when Panoptes serves subjects to be classified to the client, it includes a subject file location that the client uses to download and present to the user. How will you get the client to present and unknown subject location? erhaps your client is using our API differently to how we use it.
Ruby's not that hard (once you've used it) and you've got a fairly good grasp on the tech if you run the code in dev mode *(you are) then it's reloaded (i.e. changes appear instantly in the server) on each req and you can easily use git to revert changes, that's up to you though.
We have the work planned for the future but it's not on our radar just yet as this is how our infrastructure works. We're not trying to make a citizen science platform for every use case, just ours for the moment.
PR -> A git hub pull request for a subject storage adapter (http://en.wikipedia.org/wiki/Adapter_pattern).
Adding the aws credentials before startup fixes the error.
Hi Camallen. Thanks for all the info, it all made much more sense with the latest post. I gave my input in regards with the second paragraph to #443
So I am trying to add subjects. I use "text/plain" because really we only store coordinates and we will retrieve the images from a WMS server. (Btw I am storing the coordinates in the metadata, any better ideas?)
I get the following error in the response, but then when I list subjects they have been are added like nothing happened.
AWS::Errors::MissingCredentialsError in Api::V1::SubjectsController#create
(in detail)