Swagger returns status code 500, but the cURL command returns status code 400

matthiasschaub commented 3 months ago

Bug Description

Swagger returns status code 500, but the cURL command copied from the Swagger interface return status code 400.

General Information

Please include the following general information about the issue and list any additional steps needed to reproduce the bug.

Version of the ohsome API [e.g. 1.10.3]: 1.10.3
Which API instance was requested [e.g. remote instance at https://api.ohsome.org/v1, local instance with an h2 file or connected to the cluster, etc.]: v1
Affected endpoint(s) [e.g. /contributions/count]: /extraction
URL of your request (and request body if applicable): See below
Used HTTP method [GET or POST]: See below
Utilized tool/library for the request [e.g. cURL, Postman, ohsome-py, etc.]: cURL and Swagger

Expected Behaviour

Return an invalid filter syntax and status code 400 in Swagger as it does with cURL.

Parameters

Parameters in Swagger are the default ones except the filter.

Filter

Oneliner:

( ( highway in (primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track) or route=ferry ) and maxspeed in (5,10,15,20,25,30) ) and not ( ( footway in (separate,no) or sidewalk=separate or sidewalk:both=separate or (sidewalk:right=separate and sidewalk:left=separate) or access in (no,private,permit,military,delivery,customers) or foot in (no,private,use_sidepath,discouraged,destination) ) or ( highway in (steps,corridor) or (highway=pedestrian and not bicycle in (yes,designated) and (segregated=no or segregated=*) ) or railway=platform or highway=platform or ( highway in (footway, path) and ( ( foot in (designated,official) ) or ( footway in (access_aisle,alley,residential,link,path) or footway!=* ) and not bicycle in (yes,designated) and (segregated=no or segregated=*) ) and motor_vehicle!=yes and vehicle!=yes ) ) or ( ( highway in (footway, path) and ( foot in (yes,permissive,designated,official) or footway in (sidewalk,crossing,traffic_island,yes) ) ) or ( ( highway in (primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track) or route=ferry ) and ( sidewalk in (both,left,right,yes,lane) or sidewalk:left=yes or sidewalk:right=yes or sidewalk:both=yes or (foot in (yes,permissive,designated,official)) ) ) ) or ( ( ( ( highway in (footway, path) and ( foot in (yes,permissive,designated,official) or footway in (sidewalk,crossing,traffic_island,yes) ) ) or ( ( highway in (primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track) or route=ferry ) and ( sidewalk in (both,left,right,yes,lane) or sidewalk:left=yes or sidewalk:right=yes or sidewalk:both=yes or (foot in (yes,permissive,designated,official)) ) ) ) and ( bicycle in (yes,designated) and (segregated=no or segregated=*) ) ) or highway=pedestrian and bicycle in (yes,designated) or highway=path and motor_vehicle!=yes and vehicle!=yes ) or ( highway in (living_street,track) ) )

Muliliner:

(
     (
        highway in (
            primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track
        ) or route=ferry
    ) and maxspeed in (
        5,10,15,20,25,30
    )
) and not (
     (
        footway in (
            separate,no
        ) or sidewalk=separate or sidewalk:both=separate or (
            sidewalk:right=separate and sidewalk:left=separate
        ) or access in (
            no,private,permit,military,delivery,customers
        ) or foot in (
            no,private,use_sidepath,discouraged,destination
        )
    ) or (
        highway in (
            steps,corridor
        ) or (
            highway=pedestrian and not bicycle in (
                yes,designated
            ) and (
                segregated=no or segregated=*
            )
        ) or railway=platform or highway=platform or (
            highway in (
                footway, path
            ) and (
                 (
                    foot in (
                        designated,official
                    )
                ) or (
                    footway in (
                        access_aisle,alley,residential,link,path
                    ) or footway!=*
                ) and not bicycle in (
                    yes,designated
                ) and (
                    segregated=no or segregated=*
                )
            ) and motor_vehicle!=yes and vehicle!=yes
        )
    ) or (
         (
            highway in (
                footway, path
            ) and (
                foot in (
                    yes,permissive,designated,official
                ) or footway in (
                    sidewalk,crossing,traffic_island,yes
                )
            )
        ) or (
             (
                highway in (
                    primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track
                ) or route=ferry
            ) and (
                sidewalk in (
                    both,left,right,yes,lane
                ) or sidewalk:left=yes or sidewalk:right=yes or sidewalk:both=yes or (
                    foot in (
                        yes,permissive,designated,official
                    )
                )
            )
        )
    ) or (
         (
             (
                 (
                    highway in (
                        footway, path
                    ) and (
                        foot in (
                            yes,permissive,designated,official
                        ) or footway in (
                            sidewalk,crossing,traffic_island,yes
                        )
                    )
                ) or (
                     (
                        highway in (
                            primary,primary_link,secondary,secondary_link,tertiary,tertiary_link,road,cycleway,service,unclassified,residential,living_street,track
                        ) or route=ferry
                    ) and (
                        sidewalk in (
                            both,left,right,yes,lane
                        ) or sidewalk:left=yes or sidewalk:right=yes or sidewalk:both=yes or (
                            foot in (
                                yes,permissive,designated,official
                            )
                        )
                    )
                )
            ) and (
                bicycle in (
                    yes,designated
                ) and (
                    segregated=no or segregated=*
                )
            )
        ) or highway=pedestrian and bicycle in (
            yes,designated
        ) or highway=path and motor_vehicle!=yes and vehicle!=yes
    ) or (
        highway in (
            living_street,track
        )
    )
)

CURL

curl -X GET "https://api.ohsome.org/v1/elements/bbox?bboxes=8.67%2C49.39%2C8.71%2C49.42&clipGeometry=true&filter=(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20maxspeed%20in%20(5%2C10%2C15%2C20%2C25%2C30)%20)%20and%20not%20(%20(%20footway%20in%20(separate%2Cno)%20or%20sidewalk%3Dseparate%20or%20sidewalk%3Aboth%3Dseparate%20or%20(sidewalk%3Aright%3Dseparate%20and%20sidewalk%3Aleft%3Dseparate)%20or%20access%20in%20(no%2Cprivate%2Cpermit%2Cmilitary%2Cdelivery%2Ccustomers)%20or%20foot%20in%20(no%2Cprivate%2Cuse_sidepath%2Cdiscouraged%2Cdestination)%20)%20or%20(%20highway%20in%20(steps%2Ccorridor)%20or%20(highway%3Dpedestrian%20and%20not%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20or%20railway%3Dplatform%20or%20highway%3Dplatform%20or%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20(%20foot%20in%20(designated%2Cofficial)%20)%20or%20(%20footway%20in%20(access_aisle%2Calley%2Cresidential%2Clink%2Cpath)%20or%20footway!%3D*%20)%20and%20not%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20and%20motor_vehicle!%3Dyes%20and%20vehicle!%3Dyes%20)%20)%20or%20(%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial)%20or%20footway%20in%20(sidewalk%2Ccrossing%2Ctraffic_island%2Cyes)%20)%20)%20or%20(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20(%20sidewalk%20in%20(both%2Cleft%2Cright%2Cyes%2Clane)%20or%20sidewalk%3Aleft%3Dyes%20or%20sidewalk%3Aright%3Dyes%20or%20sidewalk%3Aboth%3Dyes%20or%20(foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial))%20)%20)%20)%20or%20(%20(%20(%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial)%20or%20footway%20in%20(sidewalk%2Ccrossing%2Ctraffic_island%2Cyes)%20)%20)%20or%20(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20(%20sidewalk%20in%20(both%2Cleft%2Cright%2Cyes%2Clane)%20or%20sidewalk%3Aleft%3Dyes%20or%20sidewalk%3Aright%3Dyes%20or%20sidewalk%3Aboth%3Dyes%20or%20(foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial))%20)%20)%20)%20and%20(%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20)%20or%20highway%3Dpedestrian%20and%20bicycle%20in%20(yes%2Cdesignated)%20or%20highway%3Dpath%20and%20motor_vehicle!%3Dyes%20and%20vehicle!%3Dyes%20)%20or%20(%20highway%20in%20(living_street%2Ctrack)%20)%20)&properties=tags&time=2016-01-01%2C2017-01-01" -H  "accept: application/json"

matthiasschaub commented 3 months ago

I am pretty sure that it is not a syntax error. This behavior happens for large filters, of which I am confident that the syntax is right. If I make those filters smaller, they work again. I am happy to give more examples and explanation.

matthiasschaub commented 3 months ago

I updated the given example.

matthiasschaub commented 3 months ago

Maybe the error message implemented in https://github.com/GIScience/oshdb/pull/531 should be return.

rtroilo commented 3 months ago

https://github.com/GIScience/ohsome-api/blob/eb421155e44b7de20d3099e133192219a84efea3/src/main/lombok/org/heigit/ohsome/ohsomeapi/inputprocessing/InputProcessingUtils.java#L410

here we need to check if the filter is "normalizable" as the oshdb-api does https://github.com/GIScience/oshdb/blob/a196cc990a75fa35841ca0908f323c3c9fc06b9a/oshdb-api/src/main/java/org/heigit/ohsome/oshdb/api/mapreducer/MapReducer.java#L1887

tyrasd commented 3 months ago

Swagger returns status code 500, but the cURL command copied from the Swagger interface return status code 400.

The 400 in the copied curl command is different from the ("expected") 500, because this query filter contains exclamation marks (!), which have special behavior in bash. The usual recommendation is to use single quotes for strings in unix shells, to avoid this problem. I don't know why swagger is using double quotes (maybe for compatibility with windows CLIs) or why it does not manually escape the ! to %21 for example.

The following curl command shows the same (bug) behavior as the main example in the original post:

curl -X GET 'https://api.ohsome.org/v1/elements/bbox?bboxes=8.67%2C49.39%2C8.71%2C49.42&clipGeometry=true&filter=(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20maxspeed%20in%20(5%2C10%2C15%2C20%2C25%2C30)%20)%20and%20not%20(%20(%20footway%20in%20(separate%2Cno)%20or%20sidewalk%3Dseparate%20or%20sidewalk%3Aboth%3Dseparate%20or%20(sidewalk%3Aright%3Dseparate%20and%20sidewalk%3Aleft%3Dseparate)%20or%20access%20in%20(no%2Cprivate%2Cpermit%2Cmilitary%2Cdelivery%2Ccustomers)%20or%20foot%20in%20(no%2Cprivate%2Cuse_sidepath%2Cdiscouraged%2Cdestination)%20)%20or%20(%20highway%20in%20(steps%2Ccorridor)%20or%20(highway%3Dpedestrian%20and%20not%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20or%20railway%3Dplatform%20or%20highway%3Dplatform%20or%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20(%20foot%20in%20(designated%2Cofficial)%20)%20or%20(%20footway%20in%20(access_aisle%2Calley%2Cresidential%2Clink%2Cpath)%20or%20footway!%3D*%20)%20and%20not%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20and%20motor_vehicle!%3Dyes%20and%20vehicle!%3Dyes%20)%20)%20or%20(%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial)%20or%20footway%20in%20(sidewalk%2Ccrossing%2Ctraffic_island%2Cyes)%20)%20)%20or%20(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20(%20sidewalk%20in%20(both%2Cleft%2Cright%2Cyes%2Clane)%20or%20sidewalk%3Aleft%3Dyes%20or%20sidewalk%3Aright%3Dyes%20or%20sidewalk%3Aboth%3Dyes%20or%20(foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial))%20)%20)%20)%20or%20(%20(%20(%20(%20highway%20in%20(footway%2C%20path)%20and%20(%20foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial)%20or%20footway%20in%20(sidewalk%2Ccrossing%2Ctraffic_island%2Cyes)%20)%20)%20or%20(%20(%20highway%20in%20(primary%2Cprimary_link%2Csecondary%2Csecondary_link%2Ctertiary%2Ctertiary_link%2Croad%2Ccycleway%2Cservice%2Cunclassified%2Cresidential%2Cliving_street%2Ctrack)%20or%20route%3Dferry%20)%20and%20(%20sidewalk%20in%20(both%2Cleft%2Cright%2Cyes%2Clane)%20or%20sidewalk%3Aleft%3Dyes%20or%20sidewalk%3Aright%3Dyes%20or%20sidewalk%3Aboth%3Dyes%20or%20(foot%20in%20(yes%2Cpermissive%2Cdesignated%2Cofficial))%20)%20)%20)%20and%20(%20bicycle%20in%20(yes%2Cdesignated)%20and%20(segregated%3Dno%20or%20segregated%3D*)%20)%20)%20or%20highway%3Dpedestrian%20and%20bicycle%20in%20(yes%2Cdesignated)%20or%20highway%3Dpath%20and%20motor_vehicle!%3Dyes%20and%20vehicle!%3Dyes%20)%20or%20(%20highway%20in%20(living_street%2Ctrack)%20)%20)&properties=tags&time=2016-01-01%2C2017-01-01' -H  "accept: application/json"

GIScience / ohsome-api