search

ArctosDB / arctos

Arctos is a museum collections management system
https://arctos.database.museum
61 stars 13 forks source link

Implement new client credentials for Google Geo API's #2058

Closed mkoo closed 5 years ago

mkoo commented 5 years ago

To ensure that we have all our background service calls up and running (and unlimited), I need to institute new client credentials for Arctos. Which means understanding the new content terms requirements then implementing.

This impacts:

Many of these have no direct Leaflet equivalent (please correct me if I'm wrong!)

On my to-do list....

dustymc commented 5 years ago

There's still some relevant stuff in https://github.com/ArctosDB/arctos/issues/2011, most critical being the decision to stay with Google or find another solution. That might be a good AWG topic.

The "background" calls keep finding ways to break other things. Currently it looks like any Agent with an Address can't be altered; we geocode them so we can do more with shipments, and the geocoder isn't happy. If I'm going to get access to a functional client_id/private_key I don't really want to dig out a bunch of complex code. If not, or not soon, I think I'm going to have to prioritize some way of fixing or disabling the background stuff I thought I could ignore.

Help?

ccicero commented 5 years ago

What are we trying to do with shipments that involves geocoding?

dustymc commented 5 years ago

Not much - there's a map shipment button from loan search results.

Origins are https://github.com/ArctosDB/arctos/issues/1325

The big question is if I can get the same type of credentials we've had from Google, in which case fixing this is about 5 seconds (literally - copy/paste/save/done!) or if I need to find another way, which (if it's even possible) would likely involve rebuilding a bunch of cryptography stuff that Google uses as a "signature."

ccicero commented 5 years ago

OK. I can see how that will be useful. Never noticed that!

Let's discuss at this morning's issue meeting.

campmlc commented 5 years ago

David Phau Manager of Developer Relations for Google Earth Engines and Google Earth Outreach was at iDigBio at Berkeley thau@google.com May be a good contact point at google for this issue and for future Arctos / Google interactions

campmlc commented 5 years ago

This issue is related to a switchover of the Arctos account from Google Business to Enterprise to something else - Berkeley Natural History Museum Project, we get in free under their quota paid for on Michelle's card Or We go to something more open source: Leaf(let)? Will do similar things but not support the services we use; not large datasets Should we start a new project with Google that is specific to Arctos We had a google enterprise account that was a grant from google that gave us business access at no cost If we do this for Arctos, then we'd need to submit a new proposal, have a contact there we can work with. Does TACC have a relationship with Google? or serious GIS in Oracle?

campmlc commented 5 years ago

Michelle says she will create a client ID and implement Friday and will make recommendations to the AWG for next week.

mkoo commented 5 years ago

FYI Dave Thau no longer is at Google!

On Thu, May 2, 2019 at 10:55 AM Mariel Campbell notifications@github.com wrote:

Michelle says she will create a client ID and implement Friday and will make recommendations to the AWG for next week.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058#issuecomment-488769822, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UJ3N5YF6SBH25ELXBLPTMTKNANCNFSM4HI4YVNA .

campmlc commented 5 years ago

Would he know someone we could contact for future reference?

atrox10 commented 5 years ago

For now, can we just disable the google agent shipping address geocoding? I can't send loans if I can't add a shipping address to an agent, which usually is a very easy task.

atrox10 commented 5 years ago

URGENT!! CAn someone fix this or just disable this function!! I can't add addresses to Agents for loans and we are at a LOAN impasse. This is the last week with some of my students doing loans, so I need to be able to add addresses to Agents!!

dustymc commented 5 years ago

I'm not sure what to do here. The address integration is causing a lot of problems.

@mkoo can I get a client_id??

Should I "temporarily" remove the geocoding??

Should I permanently remove it and the tools that use it?? (I'd rather not.)

Should I look for another solution??

HELP!!

ccicero commented 5 years ago

What's involved in temporarily removing the geocoding? I will talk with @mkoo this afternoon about the client ID, and see what the timeframe might be for doing that.

We also can't edit an existing agent address (which I was trying to do so I could use the same address, but for a different student); existing address has an "Attn:" to a person which should really be in the outside contact information. So I also can't move forward on this loan.

dustymc commented 5 years ago

I think the biggest problem with anything "temporary" would be catching back up - I think I'd have to re-geocode everything to make sure everything is current. I severely dislike the idea of 'map some arbitrary things that may have more to do with service credentials than reality' so I think we'd need to disable the shipment mapping until everything can be put back together and refreshed.

If we want to go there, I think I can disable that part of the code in an hour or so.

You can't ever alter used addresses - that would corrupt the history of shipments, the logic to prevent that is built into the DB.

I can SQL in a new address, but I think that takes us back to having inconsistent data. If it's just one I guess I could manually geocode it first....

Can we wait until you can talk to Michell before doing anything drastic?

ccicero commented 5 years ago

Yes, let's wait to talk with Michelle.

mkoo commented 5 years ago

ok this is next on my very full plate. reason #1,003 why I am not going to the retreat for s'mores!

On Tue, May 7, 2019 at 1:32 PM Carla Cicero notifications@github.com wrote:

Yes, let's wait to talk with Michelle.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058#issuecomment-490244113, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UKQLN5WYZ7SKAL7YM3PUHRMXANCNFSM4HI4YVNA .

mkoo commented 5 years ago

I faked some stuff and generated a client id and secret key for Arctos. It looks weird to me. Can you check it out in the morning, @Dusty dustymc@gmail.com please? I will be around ca. 8 or so

On Tue, May 7, 2019 at 9:28 PM Michelle S. Koo mkoo@berkeley.edu wrote:

ok this is next on my very full plate. reason #1,003 why I am not going to the retreat for s'mores!

On Tue, May 7, 2019 at 1:32 PM Carla Cicero notifications@github.com wrote:

Yes, let's wait to talk with Michelle.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058#issuecomment-490244113, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UKQLN5WYZ7SKAL7YM3PUHRMXANCNFSM4HI4YVNA .

dustymc commented 5 years ago

Still getting https://developers.google.com/maps/documentation/javascript/error-messages#invalid-client-id-map-error

It should be gme-{stuff}

mkoo commented 5 years ago

filling out a lot of forms right now... what's our monthly audience on the arctos website? can you check Google Analytics, give me an average?

On Wed, May 8, 2019 at 8:44 AM dustymc notifications@github.com wrote:

Still getting https://developers.google.com/maps/documentation/javascript/error-messages#invalid-client-id-map-error

It should be gme-{stuff}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058#issuecomment-490539940, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UKAPAFGYTOTIBK3P5LPULYPLANCNFSM4HI4YVNA .

dustymc commented 5 years ago

Looks like around 10K/m.

Screen Shot 2019-05-08 at 9 48 24 AM

but who knows how that translates to maps - I'd assume most of them see a map or two, some of them see LOTS of maps, etc.

I think firefox is recently blocking analytics by default too, just to mess with you...

mkoo commented 5 years ago

ok thanks

On Wed, May 8, 2019 at 9:51 AM dustymc notifications@github.com wrote:

Looks like around 10K/m.

[image: Screen Shot 2019-05-08 at 9 48 24 AM] https://user-images.githubusercontent.com/5720791/57392767-82707b00-7176-11e9-99d2-f671466e24af.png

but who knows how that translates to maps - I'd assume most of them see a map or two, some of them see LOTS of maps, etc.

I think firefox is recently blocking analytics by default too, just to mess with you...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058#issuecomment-490564600, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UMFKPE5CRHQGKC4A5LPUMAKNANCNFSM4HI4YVNA .

dustymc commented 5 years ago

THis is probably better - it's from https://google.secure.force.com/apex/MapsReport which I think you have access to.

Screen Shot 2019-05-08 at 9 57 37 AM
campmlc commented 5 years ago

from Michelle at AWG meeting 5-9-19 Arctos uses google maps webservices, embedded thumbnaiis but also locations, addresses, elevation lookup apis; each one hits google services and gets authenticated in different ways; google maps was easy, simple authentication key; but on the backend google has gone to pay as you go to generate new authenticate code; there is no specific quote; affects Berkeley Mapper, Symbiota; Michelle just put in her credit card, ended up being charged $8; no costs incurred over $200 because of Michelle's personal .edu account at Berkeley; could be moved under Berkeley umbrella Other option is to have client ID: The only way to get a client ID of the proper type is to go through an application process for an educational grant; or use Berkeley grant; need to know how many hits per second and overall daily use, and Arctos is way low. We can authenticate with an api key for features, elevation Can we do this for addresses? Or wait to have client ID? Michelle will answer by this afternoon

dustymc commented 5 years ago

I am trying to figure out how to use the API key instead of client ID, and I'm getting

API keys with referer restrictions cannot be used with this API.

Google says

https://developers.google.com/maps/faq#browser-keys-blocked-error

Arctos IP is

129.114.60.95

but for some reason nslookup thinks it's

129.114.52.171

so I'm not sure which of those Google would see.

@mkoo is this something that can be changed?

mkoo commented 5 years ago

Let me check that out. There may still be restrictions on using plain ol auth key with some of the geocoding api's. I nudged Google Map Key request and also campus, who did not answer my questions directly.

So no news there yet- ugh

On Mon, May 13, 2019 at 8:26 AM dustymc notifications@github.com wrote:

I am trying to figure out how to use the API key instead of client ID, and I'm getting

API keys with referer restrictions cannot be used with this API.

Google says

https://developers.google.com/maps/faq#browser-keys-blocked-error

Arctos IP is

129.114.60.95

but for some reason nslookup thinks it's

129.114.52.171

so I'm not sure which of those Google would see.

@mkoo https://github.com/mkoo is this something that can be changed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058?email_source=notifications&email_token=AATH7UNSBKFITR2POA6NT4DPVGCCZA5CNFSM4HI4YVNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVIVMQA#issuecomment-491869760, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UMT4BOXE3ZFBT34BULPVGCCZANCNFSM4HI4YVNA .

mkoo commented 5 years ago

BTW, I dont think Google matters which address they see-- the nslookup address is the dns (now served by TACC) for our ip so should redirect properly.

I just checked-- when we restrict the api key to the website it does not look at the ip address just the domain name so I dont think that matters

FYI, last 24 hours we have hit the Maps JS api (412), Places api (7), Geocoding api (1). The latter may be only one because of the limit that we currently have a request pending...

On Mon, May 13, 2019 at 8:55 AM Michelle S. Koo mkoo@berkeley.edu wrote:

Let me check that out. There may still be restrictions on using plain ol auth key with some of the geocoding api's. I nudged Google Map Key request and also campus, who did not answer my questions directly.

So no news there yet- ugh

On Mon, May 13, 2019 at 8:26 AM dustymc notifications@github.com wrote:

I am trying to figure out how to use the API key instead of client ID, and I'm getting

API keys with referer restrictions cannot be used with this API.

Google says

https://developers.google.com/maps/faq#browser-keys-blocked-error

Arctos IP is

129.114.60.95

but for some reason nslookup thinks it's

129.114.52.171

so I'm not sure which of those Google would see.

@mkoo https://github.com/mkoo is this something that can be changed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058?email_source=notifications&email_token=AATH7UNSBKFITR2POA6NT4DPVGCCZA5CNFSM4HI4YVNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVIVMQA#issuecomment-491869760, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UMT4BOXE3ZFBT34BULPVGCCZANCNFSM4HI4YVNA .

mkoo commented 5 years ago

Update: we have approval for new Google Maps Platform credits -- so all our quotas for api usage have increased or is 'limitless' so that's great!

Not so great is spending time with Google Support (a perk apparently of the new plan we're in) and basically having two separate tech support guys tell me that they would like to do a code review so we remove the client id requirements. The docs and the cloud console support the idea that all the map services we use can use the simpler api key authentication now. Working with Dusty to troubleshoot. maybe we're close....

On Mon, May 13, 2019 at 10:26 AM Michelle S. Koo mkoo@berkeley.edu wrote:

BTW, I dont think Google matters which address they see-- the nslookup address is the dns (now served by TACC) for our ip so should redirect properly.

I just checked-- when we restrict the api key to the website it does not look at the ip address just the domain name so I dont think that matters

FYI, last 24 hours we have hit the Maps JS api (412), Places api (7), Geocoding api (1). The latter may be only one because of the limit that we currently have a request pending...

On Mon, May 13, 2019 at 8:55 AM Michelle S. Koo mkoo@berkeley.edu wrote:

Let me check that out. There may still be restrictions on using plain ol auth key with some of the geocoding api's. I nudged Google Map Key request and also campus, who did not answer my questions directly.

So no news there yet- ugh

On Mon, May 13, 2019 at 8:26 AM dustymc notifications@github.com wrote:

I am trying to figure out how to use the API key instead of client ID, and I'm getting

API keys with referer restrictions cannot be used with this API.

Google says

https://developers.google.com/maps/faq#browser-keys-blocked-error

Arctos IP is

129.114.60.95

but for some reason nslookup thinks it's

129.114.52.171

so I'm not sure which of those Google would see.

@mkoo https://github.com/mkoo is this something that can be changed?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ArctosDB/arctos/issues/2058?email_source=notifications&email_token=AATH7UNSBKFITR2POA6NT4DPVGCCZA5CNFSM4HI4YVNKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVIVMQA#issuecomment-491869760, or mute the thread https://github.com/notifications/unsubscribe-auth/AATH7UMT4BOXE3ZFBT34BULPVGCCZANCNFSM4HI4YVNA .

dustymc commented 5 years ago

I think our mapping capabilities are now fully restored. Please reopen if you find something I've missed.

mkoo commented 5 years ago

Quick summary for future reference: After spending quality time with two google support staff and having Google accept our request for upgraded quota, we learned some new stuff: no longer are client id/ secret handshakes needed or will work for the API's we are using (this was the opposite of what is in the Documentation BTW). We can and should use authentication api keys that we generate. BUT also NOT in the docs is that these should be unrestricted keys (previous one was restricted for calls from our domain). Not sure I understand why but it works! Requests for any more capacity (upgrade to quota) can be made by filling out the application again and requesting more (demonstrated by our usage-- see the console).

dustymc commented 5 years ago

Also for posterity: I think I'm now piping all map/api calls through googleSignURL(), which has been modified to just assemble the URL and append the APIKey. That should make maintenance much easier, lightens the code a bit, makes it possible to temporarily disable the cache, etc.

Note that there is no encryption in this approach, and the API key apparently has to be unrestricted to authorize some calls - anyone could map something in Arctos, grab the key from their network logs, and use it. We should be monitoring usage, and Google recommends periodically pulling a new API key (which can be entered in Arctos' Global Settings).