MushroomObserver / mushroom-observer

A website for sharing observations of mushrooms.
https://mushroomobserver.org
MIT License
78 stars 25 forks source link

POST request for adding an observation #1558

Closed fran-aubry closed 1 year ago

fran-aubry commented 1 year ago

It looks like the documentation for adding an observation is incomplete. Could you please help me figure out the proper format of the URL?

Namely:

  1. The example url has a parameter named "name" while the online form request a "scientific name". Therefore, it is not obvious how each of the fields in the form should be named in the url. Could you provide a complete mapping?

  2. For fields like "Used references" it is unclear what should be sent. Both a string and a boolean if checked? Just a string?

  3. The readme refers to a file script/test_api to see how images should be encoded. However it seems that this file doesn't exist.

  4. I've made a few requests using my key in this url:

    https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the my observations page.

Thank you in advance for you help.

Best regards.

pellaea commented 1 year ago

Hi there,

The variable "name" in a POST request to observations is the scientific name. To get a full list of parameters do:

curl -X POST -F api_key=___ -F help=1 https://mushroomobserver.org/api2/observations

Here they are in the order they appear in the online form, with some extra help notes:

date: date (YYYY-MM-DD) location: place_name (e.g., "Pasadena, California, USA") is_collection_location: boolean (default, is this location where mushroom was found?) latitude: latitude longitude: longitude altitude: altitude gps_hidden: boolean (default=false, hide exact coordinates?) name: name (default=Fungi Bartl.) vote: float (default 3="I'd call it that", 2="probably", 1="could be", etc.) reason_1: string (recognized by sight, just send a string, if you include this parameter it's the same as checking the box, even if you pass an empty string) reason_2: string (used references) reason_3: string (micro features) reason_4: string (chemical features) has_specimen: boolean collectors_name: string collection_number: string herbarium: herbarium accession_number: string (unique fungarium id) notes: string notes[$field]: string (set value of the custom notes template field, substitute field name for "$field")

Other parameters you probably won't use:

images: image list (ids of images you have already uploaded, it's easier to create the obs first then upload images one at a time attaching each one to the obs as you go) thumbnail: image (id of one of the images you have already attached, this is really only used in PATCH requests) species_lists: species_list list (id, must have edit permission) projects: project list (id, must be member) initial_det: string (ignore this) source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website, default=mo_api) log: boolean (default, log this action on main page activity log and RSS feed?)

If sorry, yes, it looks like script/test_api has disappeared.

It looks like you can do uploads like this:

curl -X POST -F @.*** -F etc. https://mushroomobserver.org/api2/images

Let me know if that doesn't work.

I tried your test URL verbatim (replacing xxx with my api key) and it worked just fine. Can you give me your user id, and tell me approximately when you tried it, and I'll look it up in the log to see if I can figure out what went awry.

Did I miss anything?

-Jason

On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is incomplete. Could you please help me figure out the proper format of the URL?

Namely:

1.

The example url has a parameter named "name" while the online form https://mushroomobserver.org/observations/new request a "scientific name". Therefore, it is not obvious how each of the fields in the form should be named in the url. Could you provide a complete mapping? 2.

For fields like "Used references" it is unclear what should be sent. Both a string and a boolean if checked? Just a string? 3.

The readme refers to a file script/test_api to see how images should be encoded. However it seems that this file doesn't exist. 4.

I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the my observations page.

Thank you in advance for you help.

Best regards.

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

fran-aubry commented 1 year ago

Hey, thank you for the reply.

I made this request today at around 11:02 (UTC+8)

https://mushroomobserver.org/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes

I got a 200 response but it doesn't show in my observation list. Could you take a look?

Also, is there a proper way to test these requests? I don't want to pollute the database while testing.

On Thu, Aug 17, 2023 at 2:44 PM Jason Hollinger @.***> wrote:

Hi there,

The variable "name" in a POST request to observations is the scientific name. To get a full list of parameters do:

curl -X POST -F api_key=___ -F help=1 https://mushroomobserver.org/api2/observations

Here they are in the order they appear in the online form, with some extra help notes:

date: date (YYYY-MM-DD) location: place_name (e.g., "Pasadena, California, USA") is_collection_location: boolean (default, is this location where mushroom was found?) latitude: latitude longitude: longitude altitude: altitude gps_hidden: boolean (default=false, hide exact coordinates?) name: name (default=Fungi Bartl.) vote: float (default 3="I'd call it that", 2="probably", 1="could be", etc.) reason_1: string (recognized by sight, just send a string, if you include this parameter it's the same as checking the box, even if you pass an empty string) reason_2: string (used references) reason_3: string (micro features) reason_4: string (chemical features) has_specimen: boolean collectors_name: string collection_number: string herbarium: herbarium accession_number: string (unique fungarium id) notes: string notes[$field]: string (set value of the custom notes template field, substitute field name for "$field")

Other parameters you probably won't use:

images: image list (ids of images you have already uploaded, it's easier to create the obs first then upload images one at a time attaching each one to the obs as you go) thumbnail: image (id of one of the images you have already attached, this is really only used in PATCH requests) species_lists: species_list list (id, must have edit permission) projects: project list (id, must be member) initial_det: string (ignore this) source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website, default=mo_api) log: boolean (default, log this action on main page activity log and RSS feed?)

If sorry, yes, it looks like script/test_api has disappeared.

It looks like you can do uploads like this:

curl -X POST -F @.*** -F etc. https://mushroomobserver.org/api2/images

Let me know if that doesn't work.

I tried your test URL verbatim (replacing xxx with my api key) and it worked just fine. Can you give me your user id, and tell me approximately when you tried it, and I'll look it up in the log to see if I can figure out what went awry.

Did I miss anything?

-Jason

On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is incomplete. Could you please help me figure out the proper format of the URL?

Namely:

1.

The example url has a parameter named "name" while the online form https://mushroomobserver.org/observations/new request a "scientific name". Therefore, it is not obvious how each of the fields in the form should be named in the url. Could you provide a complete mapping? 2.

For fields like "Used references" it is unclear what should be sent. Both a string and a boolean if checked? Just a string? 3.

The readme refers to a file script/test_api to see how images should be encoded. However it seems that this file doesn't exist. 4.

I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the my observations page.

Thank you in advance for you help.

Best regards.

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE>

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1681714192, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2CLA6YXYM6OVQR6EVT4MDXVW4UHANCNFSM6AAAAAA3TRHENE . You are receiving this because you authored the thread.Message ID: @.***>

pellaea commented 1 year ago

Hi again,

For some reason your requests are coming through as HEAD requests, not POSTs.

HEAD "/api2/observations" for 51.15.164.30 at 2023-08-18 03:10:53 +0000 HEAD "/observations/new" for 51.15.164.30 at 2023-08-18 03:10:55 +0000 HEAD "/api2/images" for 51.15.164.30 at 2023-08-18 03:10:55 +0000 HEAD "/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes" for 51.15.164.30 at 2023-08-18 03:10:57 +0000 HEAD "/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn" for 51.15.164.30 at 2023-08-18 03:10:58 +0000 HEAD "/account/login/new" for 51.15.164.30 at 2023-08-18 03:10:58 +0000

That's all of the requests from that IP today.

I wouldn't worry overly about polluting the database. You can only create one object at a time. Just test your code on a single observation at a time until it looks like all the kinks are worked out, then let it loose. You can easily delete anything on the website as necessary. And more to the point, there isn't any other option!

-Jason

On Thu, Aug 17, 2023 at 11:06 PM fran-aubry @.***> wrote:

Hey, thank you for the reply.

I made this request today at around 11:02 (UTC+8)

https://mushroomobserver.org/api2/observations?api_key=****&date=2023-08-18&place_name=Earth&latitude=20.401271959729456&longitude=-3.33984375&altitude=288.0030212402344&gps_hidden=false&vote=0&reason_1=Yes

I got a 200 response but it doesn't show in my observation list. Could you take a look?

Also, is there a proper way to test these requests? I don't want to pollute the database while testing.

On Thu, Aug 17, 2023 at 2:44 PM Jason Hollinger @.***> wrote:

Hi there,

The variable "name" in a POST request to observations is the scientific name. To get a full list of parameters do:

curl -X POST -F api_key=___ -F help=1 https://mushroomobserver.org/api2/observations

Here they are in the order they appear in the online form, with some extra help notes:

date: date (YYYY-MM-DD) location: place_name (e.g., "Pasadena, California, USA") is_collection_location: boolean (default, is this location where mushroom was found?) latitude: latitude longitude: longitude altitude: altitude gps_hidden: boolean (default=false, hide exact coordinates?) name: name (default=Fungi Bartl.) vote: float (default 3="I'd call it that", 2="probably", 1="could be", etc.) reason_1: string (recognized by sight, just send a string, if you include this parameter it's the same as checking the box, even if you pass an empty string) reason_2: string (used references) reason_3: string (micro features) reason_4: string (chemical features) has_specimen: boolean collectors_name: string collection_number: string herbarium: herbarium accession_number: string (unique fungarium id) notes: string notes[$field]: string (set value of the custom notes template field, substitute field name for "$field")

Other parameters you probably won't use:

images: image list (ids of images you have already uploaded, it's easier to create the obs first then upload images one at a time attaching each one to the obs as you go) thumbnail: image (id of one of the images you have already attached, this is really only used in PATCH requests) species_lists: species_list list (id, must have edit permission) projects: project list (id, must be member) initial_det: string (ignore this) source: enum (limit=mo_android_app|mo_api|mo_iphone_app|mo_website, default=mo_api) log: boolean (default, log this action on main page activity log and RSS feed?)

If sorry, yes, it looks like script/test_api has disappeared.

It looks like you can do uploads like this:

curl -X POST -F @.*** -F etc. https://mushroomobserver.org/api2/images

Let me know if that doesn't work.

I tried your test URL verbatim (replacing xxx with my api key) and it worked just fine. Can you give me your user id, and tell me approximately when you tried it, and I'll look it up in the log to see if I can figure out what went awry.

Did I miss anything?

-Jason

On Thu, Aug 17, 2023 at 1:54 AM fran-aubry @.***> wrote:

It looks like the documentation for adding an observation is incomplete. Could you please help me figure out the proper format of the URL?

Namely:

1.

The example url has a parameter named "name" while the online form https://mushroomobserver.org/observations/new request a "scientific name". Therefore, it is not obvious how each of the fields in the form should be named in the url. Could you provide a complete mapping? 2.

For fields like "Used references" it is unclear what should be sent. Both a string and a boolean if checked? Just a string? 3.

The readme refers to a file script/test_api to see how images should be encoded. However it seems that this file doesn't exist. 4.

I've made a few requests using my key in this url:

https://mushroomobserver.org/api2/observations?api_key=xxx&name=Agaricus&location=Pasadena,+California,+USA&date=2016-08-06&notes=growing+in+lawn

I got a 200 on each of them. However I don't see them when I go to the my observations page.

Thank you in advance for you help.

Best regards.

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAYTNNMIKEPNU6EBVIDCL73XVWWZDANCNFSM6AAAAAA3TRHENE>

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1681714192>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AA2CLA6YXYM6OVQR6EVT4MDXVW4UHANCNFSM6AAAAAA3TRHENE>

. You are receiving this because you authored the thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1683270594, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNLN3ZRPF424AFX2Q2TXV3LZRANCNFSM6AAAAAA3TRHENE . You are receiving this because you commented.Message ID: @.***>

fran-aubry commented 1 year ago

Thanks for the help. I manage to fix the observation creation process. I'm looking into image uploads now but I keep getting a 500 error. This is what I'm doing:

curl -X POST -F api_key='...' -F upload_url='https://media.istockphoto.com/id/1460769162/photo/two-mushrooms-in-a-dark-forest.jpg?s=2048x2048&w=is&k=20&c=Z8kbfYrgfGd4rUdkCVg9pt-d6_Tn2dF0VBSUuHhQ21I=' 'https://mushroomobserver.org/api2/images'

I suspect it is because some other parameters might be required?

I see the list here but it doesn't specify which ones are required:

Usage: copyright_holder: string (limit=100 chars); date: date (when photo taken); license: license; md5sum: string; notes: string; observations: observation list (must have edit permission); original_name: string (limit=120 chars, original file name or other private identifier); projects: project list (must be member); upload: upload; upload_file: string; upload_url: string; vote: enum (limit=1|2|3|4)

Could you take a look to see what I'm doing wrong with my request?

Thank you in advance :pray:

pellaea commented 1 year ago

Great news that you've gotten your end working! Now I gotta get our end working. :) Weird, though, because I've used this and our mobile app uses this, and it works fine for us. And in fact it is actually uploading the image. See:

https://mushroomobserver.org/images/1591185 https://mushroomobserver.org/images/1591186 https://mushroomobserver.org/images/1591187 https://mushroomobserver.org/images/1591188

It should be returning the image id in the result, but for some reason the ids are nil which is causing an internal error. I'm going to have to look into this more. I don't think it has anything to do with you. Sorry about that! I'll get back to you tomorrow.

-Jason

On Mon, Aug 21, 2023 at 12:43 AM fran-aubry @.***> wrote:

Thanks for the help. I manage to fix the observation creation process. I'm looking into image uploads now but I keep getting a 500 error. This is what I'm doing:

curl -X POST -F api_key='...' -F upload_url=' https://media.istockphoto.com/id/1460769162/photo/two-mushrooms-in-a-dark-forest.jpg?s=2048x2048&w=is&k=20&c=Z8kbfYrgfGd4rUdkCVg9pt-d6_Tn2dF0VBSUuHhQ21I=' 'https://mushroomobserver.org/api2/images'

I suspect it is because some other parameters might be required?

I see the list here but it doesn't specify which ones are required:

Usage: copyright_holder: string (limit=100 chars); date: date (when photo taken); license: license; md5sum: string; notes: string; observations: observation list (must have edit permission); original_name: string (limit=120 chars, original file name or other private identifier); projects: project list (must be member); upload: upload; upload_file: string; upload_url: string; vote: enum (limit=1|2|3|4)

Could you take a look to see what I'm doing wrong with my request?

Thank you in advance 🙏

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1685631875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNPQSGS3AJFYDP4SCALXWLRNNANCNFSM6AAAAAA3TRHENE . You are receiving this because you commented.Message ID: @.***>

fran-aubry commented 1 year ago

Thank you for looking into it.

Do you have an idea of when this will be fixed?

pellaea commented 1 year ago

Thank you so much for reminding me. I totally spaced on that. Shoot, and I'm going to be out for a few days. I'm so sorry. I wrote it on the whiteboard in big caps so I won't forget!

On Thu, Aug 24, 2023 at 4:48 AM fran-aubry @.***> wrote:

Thank you for looking into it.

Do you have an idea of when this will be fixed?

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1691272595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNLPJKC4IQL3UXBRVZLXW4IMXANCNFSM6AAAAAA3TRHENE . You are receiving this because you commented.Message ID: @.***>

pellaea commented 1 year ago

Hi Fran,

I had a minute, so I took a look and have at least narrowed down the problem. It's the "upload by URL" feature that's failing. I guess no one has ever tried to use that feature before, and it must not be (effectively) unit tested. If you download the image yourself, then upload it as a file instead of URL:

curl -X POST -F api_key='...' -F @.***' ' https://mushroomobserver.org/api2/images'

That works fine. Yeah, the API can do it, but the controller hasn't actually "hooked it up". I'll take a look at this later.

-Jason

On Thu, Aug 24, 2023 at 10:26 AM Jason Hollinger @.***> wrote:

Thank you so much for reminding me. I totally spaced on that. Shoot, and I'm going to be out for a few days. I'm so sorry. I wrote it on the whiteboard in big caps so I won't forget!

On Thu, Aug 24, 2023 at 4:48 AM fran-aubry @.***> wrote:

Thank you for looking into it.

Do you have an idea of when this will be fixed?

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1691272595, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNLPJKC4IQL3UXBRVZLXW4IMXANCNFSM6AAAAAA3TRHENE . You are receiving this because you commented.Message ID: @.***>

fran-aubry commented 1 year ago

All is working now. URLs would have been more convenient but it's good enough for now.

Thank you for your time :pray:

pellaea commented 1 year ago

Oh good. I dove in deeper, and it should actually work. And in fact we do even have unit tests for it. But the error messages are being swallowed and hidden.

On Fri, Aug 25, 2023 at 6:20 AM fran-aubry @.***> wrote:

All is working now. URLs would have been more convenient but it's good enough for now.

Thank you for your time 🙏

— Reply to this email directly, view it on GitHub https://github.com/MushroomObserver/mushroom-observer/issues/1558#issuecomment-1693134146, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAYTNNL44H6DKFHYGHMVAD3XXB36LANCNFSM6AAAAAA3TRHENE . You are receiving this because you commented.Message ID: @.***>