Http Referer on calls to ODH

[RudiThoeni]

Hi

We have a logging for all ODH calls where we group the calls by referer. I tested with http://journey.opendatahub.bz.it and i have seen that the http referer send to ODH is:

json.output.referer | https://tourism.opendatahub.bz.it/api/ODHActivityPoi/?searchfilter=partsc&pagesize=5&language=de&poitype=447&active=true&fields=Id,Detail.de.Title,GpsInfo

Strange that the whole call destination is sent as HTTP Header Referer..... Maybe the framework is doing it automatically...... the referer normally should be the webpage where the call starts......

Is it possible to set an explicit HTTP Referer Header which only passes the host from which the call is made ?at example (journey.opendatahub.bz.it, or mobility.meran.eu) that would be perfect so we can make advanced statistics / dashboards

thx and cheers Rudi

[stefanocudini]

you are right! I'll fix this

[stefanocudini]

I don't understand, trying to replicate the problem but it doesn't seem to me that the client is sending anything other than hostname: and in background the geocoder service only sends this http header to the ODH api: "User-Agent": "OpenMove-Geocoder-Client"

[RudiThoeni]

very strange, i get the correct User-Agent in my logs but the referer is not right.....

As you said The geocoder service does not pass the referer..... If it's not possible to fix/config on the geocoder side maybe i cann add a parameter which overwrites the referer header........

[RudiThoeni]

What i mean is the call to odh is made from https://geocoder.otp.opendatahub.testingmachine.eu/v1/autocomplete?text=partschin&lang=&boundary.rect.min_lat=46.18828&boundary.rect.min_lon=10.470121&boundary.rect.max_lat=47.08878&boundary.rect.max_lon=12.255011 the referer is passed from journey -> geocoder but not from geocoder --> odh.......... Is it possible to (re-)set it manually on geocoder backend? another option could be i add a parameter to the odh call which can be filled with the referer got from journey - geocoder ....... The interesting statistics for us would be to know if the user was on journey, or on mobility.meran etc.... what did he searched on odh etc..

[stefanocudini]

currently the geocoder service does not send any additional headers other than the user-agent. I think it is the http proxy that appends these headers.

what comes out of the geocoder towards api or dh are the following headers:

$ nc -l -k -p 9090
GET /api/ODHActivityPoi/?searchfilter=cose&pagesize=5&language=en&poitype=447&active=true&fields=Id,Detail.en.Title,GpsInfo HTTP / 1.1
User-Agent: OpenMove-Geocoder-Client
host: localhost: 9090
content-length: 0
Connection: close

GET /api/Accommodation?searchfilter=hotel&pagesize=5&language=en&poitype=447&active=true&fields=Id,AccoDetail.en.Name,Latitude,Longitude HTTP / 1.1
User-Agent: OpenMove-Geocoder-Client
host: localhost: 9090
content-length: 0
Connection: close

GET /api/ODHActivityPoi/?searchfilter=hotel&pagesize=5&language=en&poitype=447&active=true&fields=Id,Detail.en.Title,GpsInfo HTTP / 1.1
User-Agent: OpenMove-Geocoder-Client
host: localhost: 9090
content-length: 0
Connection: close

To verify this I'm capturing the requests coming out of the geocoder with netcat in locale listeningnc -l -k -p 9090 And Referer header do not appear in my local environment

[stefanocudini]

however we can config custom header in geocoder request here: https://github.com/openmove/odh-mentor-otp/blob/master/geocoder/config.yml#L16

but I think that http proxy may be overwrite as it is currently doing.

[RudiThoeni]

Sorry, I overlooked that the requests are done to /api/ . ODH here does a redirect to /v1/ so i think the referer is overwritten/set by the redirect ;)

We can try the following Set the ODH Endpoints from https://tourism.opendatahub.bz.it/api/ODHActivityPoi/ to https://tourism.opendatahub.bz.it/v1/ODHActivityPoi/ and we set a manually a referer like "https://journey.opendatahub.bz.it/" and it should work! When possible to set it by Environment variables would be perfect, so i can set another referer for testing + prod environment

thx and have a good weekend!

[RudiThoeni]

ok i set the ODH Endpoints to "v1" now no referer is passed

so let's set manually the referer header as environment variable next week

[stefanocudini]

I sent this PR that inslude Google Analytics integration #130 I think you can also use this tool to track referrers

[RudiThoeni]

Hi,

ok i integrate the latest PR. Is it possible to add a header (Referer) here https://github.com/openmove/odh-mentor-otp/blob/master/geocoder/config.yml#L16 so i can set the Referer via environment variable.... Sorry i have to ask you because i don`t know how the config format is, i guess something like:

headers: Referer https://journey.opendatahub.bz.it

[RudiThoeni]

hi @stefanocudini any news about how to set this headers config? thx

[stefanocudini]

@RudiThoeni sent a new PR about this here: https://github.com/noi-techpark/odh-mentor-otp/pull/132

[RudiThoeni]

thx!

[RudiThoeni]

working!