MAJOR PROBLEM when refreshing Data Sources under Import & Settings

[Area48Webmaster]

When I hit the refresh under Import & Settings, which I do every night, tonight I got this response to all 1,160 meetings and we currently have nothing listed on the website. Please help!

_Google gave an unexpected response for address 362 State St., Albany, NY, 12210, United States. Response was (object) array( 'error_message' => 'This IP, site or mobile application is not authorized to use this API key. Request received from IP address 35.208.14.103, with empty referer', 'results' => array ( ), 'status' => 'REQUEST_DENIED', )_

[arbliss]

same issue for me as well. doesn't matter whether i supply the plugin with a mapbox api key or a google maps api key.

[joshc47]

Area 47 is also experiencing this issue. We are not nor have ever used Google Maps

[TheGreenSourcerer]

The documentation states that:

We get our geocoding positions from Google (this true even if your maps are by Mapbox).

so the error message seems to have much sense.

[TheGreenSourcerer]

@tech2serve what's the reason for using a common Google API key for all geocoding requests? https://github.com/code4recovery/12-step-meeting-list/blob/master/includes/functions.php#L585

[tech2serve]

Hey y'all. First off, I do apologize for the inconvenience, but I think you'll understand when I explain this. We have been using a single API key since the very beginning, but something (we're still trying to track down for sure the cause) has caused our geocodes to spike big time, so much so that that little bit of money set aside was exhausted and now there is a several hundred dollar bill from Google. We're not set up to pay this (but we will obviously).

In the meantime, some restrictions were added to the key to ensure we don't end up owing even more money. We are trying to clear this issue quickly. Some of our volunteers are already putting together potential patches.

If you're comfortable with the code, and you can set up your own Google API key, you can change line 585 of the functions.php file to your own API key. This will get you up and running. Long term, any update will overwrite this, but that update should contain the fix.

More to follow soon.

[joshc47]

Thanks for the update. Can you let us know how we can donate to help cover your expenses?

[arbliss]

thanks for the help! could the usage have spiked because the api key was visible on github, and used by others?

[TheGreenSourcerer]

Thank you for the explanation. However, maybe a better long-term solution would be to allow override the API key with a value provided by the plug-in user? I guess that many sites are using the Google API anyway. This would also (at least in my opinion) be a more 7th-tradition-friendly solution 🙂

[tech2serve]

thanks for the help! could the usage have spiked because the api key was visible on github, and used by others?

That is one of the possibilities. We've had this on the back burner as something to fix.

Thanks for the update. Can you let us know how we can donate to help cover your expenses?

Thank you. We'll discuss this generous offer.

One of the reasons we hadn't solved this earlier is because we know a lot of users aren't comfortable, or aren't able, to put a credit card on file with Google. However, as @TheGreenSourcerer has correctly pointed out, we should have made it so those who are using a Google key would have it be used in place of our key.

Again, we'll strive to get this resolved ASAP.

[OSAwebsite]

Same issue herre in Ontario South Canada. I do not even use Google. I use mapbox. Can you provide us with a generic code for Google so that this does not get in our way??

[OSAwebsite]

MAJOR ISSUE indeed because NONE of my meeting pages is now updating. I tried reimporting one that is urgent, for tonight, and when that did not work, duplicated another meeting and manually edited to reflect the meeting I need to change and that also does not work. Please provide a workaround ASAP until this is fixed.

[tech2serve]

Please provide a workaround ASAP until this is fixed.

I did provide a workaround above that involves you establishing your own Google API key, and then replacing ours with it until we get a more permanent solution in place.

[OSAwebsite]

The thing is that I do NOT use Google Maps. I use Mapbox. There has never been any api key in the plugin settings for the Google API key. I tried recreating the meeting as a new meeting, manually newly added to the plugin with NO address. It pretends to accept this, I can preview the page but it is not visible in the frontend. Again: I need some way to be able to update my pages. ASAP.

[OSAwebsite]

It is not that easy to set up my own API key either. I will have to pay Google for that, it costs more than a couple of hours for this to be implemented. Besides, I am NOT using this API so why is that breaking?

[joshc47]

OSAwebsite, this is a free addon that you are not paying for and are in no position to be making demands from the volunteer developer. The issue has been explained and a workaround presented, if you are not capable of implementing it I suggest you attempt to engage someone who is rather than yell at tech2serve, who does great service for our community.

[97eighty]

@OSAwebsite I'm working on a fix to support user installed map keys.

[OSAwebsite]

Josh, I am sorry it if looked like yelling. I am not yelling. Just trying to emphasize my urgency. Tech2serve has no reason to doubt how grateful I am about the tool. I too am a volunteer in the program. Which does not mean that people do not come to expect certain things from me.

So, let's approach this the program way. The only change I have recently made is upgrade to Wordpress 5.6. Is it possible this triggered this issue?

How can I, as the user of this otherwise fabulous plug-in, be expected to understand that issues with the Google API key are at the root of the issue, when I am not even using Google?

When I looked at the front end of the meeting pages, all of my meetings now show an error on the details page, in the maps section and it shows as a Google Maps issue. That makes no sense to me because, as I said, I never ever entered a Google API key. Ever.

I now have a notice on my front page about this error and I will list changes to meetings there for the time being.

In this era of Covid online meetings have become hugely important in our 12 step work. To reach the families and friends of alcoholics we need to be able to trust the credentials on the website. I can do without the mapview in the details, but I cannot do without the ability to edit and republish.

Please keep us posted about workarounds.

Nicky

[OSAwebsite]

97eighty: thank you. Good luck.

[tech2serve]

It is not that easy to set up my own API key either. I will have to pay Google for that, it costs more than a couple of hours for this to be implemented. Besides, I am NOT using this API so why is that breaking?

Hi @OSAwebsite,

For the ease of our users, you've hit exactly on the reason why we did it this way: so people didn't have to set up their own Google API Key. Unfortunately, things changed and we need to address it. None of us can afford to pay a several hundred dollar bill every month for the entire community. Spreading this out among users will likely make it so nobody pays because individually, everybody stays under the limit.

I'm sensing a couple of misconceptions from your comment.

1. The Google API key in question is used for geocoding (i.e., making sure the address entered is valid). We added the option of using Mapbox for mapping to reduce the Google usage, and this worked for a couple of years. However, even if a website is using Mapbox for mapping, our Google API key is still used for geocoding. That is what is causing us collectively to go over Google's free limit. And I believe that every time a website imports the complete meeting list, more geocodings are conducted. This is another reason I wish users would avoid doing repeated imports of meeting lists (especially deleting the database each time). But, some do, and I can't control this (echos of the Serenity Prayer).
2. Having you (and other webmasters) establish a Google API key makes it so it is unlikely anybody will pay Google. It is unlikely an individual website will go over the limits and incur an actual monthly charge. However, if we continue to use a single API key, we will likely go over the limit. This is not sustainable.

We're also looking at using Mapbox for geocoding for those who have Mapbox keys. This will require us to write and test code to do the geocodes. And we're not sure how accurate Mapbox geocoding will be compared to Google. Lots of unknowns here.

The process to set up a Google API key is not insurmountable. It may take a little time, but quite possibly not two hours. Or maybe it will depending on your comfort level with Google's interface. Others have done this, and you might get some help on TIAA Forums.

In any case, we hope to have a solution out soon. And in the meantime, the workaround I provided remains the best short term fix.

[OSAwebsite]

Thanks, Tech2Serve, for the explanation . Now, here is the weird thing I am seeing: I suddenly see a Google API key in the plugin's settings. I was just in the process of rechecking my own documentation because I remembered getting this key when I just started out with the plugin and the building of the website (I am a firsttimer with that...) As I was copying it and wanting to paste it, I noticed it is now there and I now do not see a mapbox key. I have one, though, from awhile back. So I will try and paste that in to see if it helps. It's a bit of a mystery to me.

It might be simpler for the development team to forego the geocoding and leave the setting up of the map to the users of the plugin. That puts the responsibility where it should be, especially when there is moneys involved. Your team gives us a lot of features that we are all happy with, and the showing of the map was (for me and certainly during Covid) the icing on the cake and not essential. People could always copy the meeting location address and paste it in their own web browser, after all. That too puts the responsibility and action where it should be: with the people who want and need to find us.

So, if possible, would that be something the team would consider? Take out the link between the actual map location and the entered location details in the plugin? That will give the team to sort this out until something is figured out - paid or unpaid.

In closing: do you think that pasting the Mapbox token is OK? or is the details meeting page map view linked to Google maps whether I have the Mapbox token or not?

Thanks again for your reply. I can tell you have a lot of recovery - and serenity! And hopefully a happy new year as well!

[OSAwebsite]

I also had a look at the suggested workaround in the functions.php and it does not have line 585. To be honest: I am not comfortable fiddling with those kinds of settings because I just lack that kind of technical knowledge of Wordpress. Also, I have to keep in mind that too much customisations will make it hard to rotate my service to my successor.

I did try to use the Google API key that I used to use. It does not solve the issues. I then put in the Mapbox API key instead. It does not solve the issues.

I have an entry (new, created within Wordpress) for the meeting that needed changes, but I cannot get it out of draft mode. It is the publishing step that is not activated. The Update button does not become active. When I happen to get it to show, and click it, nothing happens. It does not make a difference whether the meeting is new or edited or re-imported. it does not matter if there is a location entered or not. I am just stuck.

As for the importing: I have not done that for a long time. I keep my own files up to date, then jump to the meeting list page and make my edits there. If all of my meetings were to massively jump from online back to physical meetings, I will do it though, because it just takes to much time to make those types of batch changes manually.

Tonight I wanted to check if I could solve the issue by doing a single-meeting import, and deleting the old entry. But it did not work so then meeting is not present at all in the plugin listings.

[tech2serve]

Thanks, Tech2Serve, for the explanation . Now, here is the weird thing I am seeing: I suddenly see a Google API key in the plugin's settings. I was just in the process of rechecking my own documentation because I remembered getting this key when I just started out with the plugin and the building of the website (I am a firsttimer with that...) As I was copying it and wanting to paste it, I noticed it is now there and I now do not see a mapbox key. I have one, though, from awhile back. So I will try and paste that in to see if it helps. It's a bit of a mystery to me.

"There can be only one!" (if you're old enough to remember the 1986 Highlander :-) ). The plugin can be configured to use Mapbox OR Google for mapping, but not both. So, if you add a Google API key into the settings, anything in the Mapbox field gets removed. And vice versa.

It might be simpler for the development team to forego the geocoding and leave the setting up of the map to the users of the plugin. That puts the responsibility where it should be, especially when there is moneys involved. Your team gives us a lot of features that we are all happy with, and the showing of the map was (for me and certainly during Covid) the icing on the cake and not essential. People could always copy the meeting location address and paste it in their own web browser, after all. That too puts the responsibility and action where it should be: with the people who want and need to find us.

So, if possible, would that be something the team would consider?

No, for two reasons primarily. First, we need to ensure accurate addresses in order to support providing directions for the user. Rewriting this logic would be a step backwards for the plugin, and would likely take more time than getting us to a long-term solution. Second, the majority of our sites feed into the Meeting Guide app architecture. Meeting Guide also requires a valid (i.e., geocoded) address.

Take out the link between the actual map location and the entered location details in the plugin? That will give the team to sort this out until something is figured out - paid or unpaid.

In closing: do you think that pasting the Mapbox token is OK? or is the details meeting page map view linked to Google maps whether I have the Mapbox token or not?

I'm not entirely sure I'm understanding what you're asking here. You cannot interchange the Mapbox and Google keys either in the settings page or in the functions.php (the latter must be a Google API key for geocoding.

Thanks again for your reply. I can tell you have a lot of recovery - and serenity! And hopefully a happy new year as well!

And to you as well.

Tim

[OSAwebsite]

What I was trying to say is this:

I originally had two API keys. One for Google, one for Mapbox. I don 't remember if the Google one was the one provided by the plugin upon first installation. I think that it was.

Tonight, when I was starting to have issues, I noticed in the import & settings section that in the plugin I did not have a Google token - just a Mapbox one. Which made sense to me because I remember choosing to use Mapbox instead.

To find out if my Google Maps token would solve the issue I put that in. No luck. So I put the Mapbox one back in. Issues remain the same.

So I am now looking at the settings and only have the Google Maps API key. Won't paste it here in full but it starts with AIza. Not sure if that is the original one that came with the first installation of the plugin.

Make sense?

[tech2serve]

Yes. You'll need to create your own Google API key for geocoding. You can also then use it for mapping.

The first hot fix we're working on implements a solution whereby the Google API key the user enters in the Settings page will also be used for geocoding (and mapping).

[OSAwebsite]

OK, I will wait for the hot fix. If action is required by webmasters, I would very much appreciate some guidance about where to go and what to choose. From what I can tell the Google library has a massive number of options. No idea how to choose and what would be according to your best practices. I am sure I am not the only volunteer with limited knowledge about these things and the last thing I want to do is make a choice that results in high invoices that I did not anticipate.

Have a good night and wish the team good luck with this.

Thanks, Nicky

[HowardL-md]

OSAwebsite, The line 585 referred to above is is at /wp-content/plugins/12-step-meeting-list/includes/functions.php line 585: 'key' => 'xxxxxxxxxxxxxxxx', Put your key where I have the x's above.

That being said, tech2serve, is it really necessary to use that geolocate api when it puts you this kind of situation with the g'ster? Personally, I am a fan of no affiliation. If I enter an incorrect address it's my bad and I don't expect you to confirm my entries.

[OSAwebsite]

Hi Howard,

I was thinking the same thing earlier. But I think the issue is more complex although I obviously do not have all the ins and out or even the understanding. It is not my place to explain what is going on, but here is where I believe the bottle necks are:

Several conundrums: -the meeting plugin started out as just that: a helpful tool to show meetings on websites -the plugin was made available for free, forever, to all 12 step programs. Which was met with loud applause and cheers! -the plugin then could be configured to share information about meetings between (AA only) websites -the plugin next became a conduit to feed the accompanying and AA specific and proprietory (phone) app -for both of the previous 2, the team must have wanted a way to check that meeting addresses were entered correctly.

Remember that this app is a tool that is for the whole of AA - which means a whole lot of meetings in several countries. As developing team you want to find a way to prevent duplicate locations for the same meeting because of a typo of some sort. Or have a situation where Google cannot plot how to get you the directions to get there.

The check on the address, I believe, under the hood is done with one of Google's Geocode APIs. I know it happens at the moment of saving a new listing and when using the import feature. It feels as if editing meetings through the plugin also triggers it, even if the editing has nothing to do with the address. It's like the complete listing is put through a number of checks. That would make sense to me.

What we are seeing now looks like a domino effect that only started with unpaid bills and interruption of service by Google.

If you do not pay your (unexpected) bills for the number of API calls, Google will no longer allow vetting of addresses. If you set up your coding to not allow any updates to a listing when vetting fails, everything is now stalled. Which is where I am with my meetings. Stuck in draft, not able to publish or update in any way.

My staging site has an older version of the TSML plugin and Wordpress; yet, it shows the same issue. So the root of the problem is not in recent plugin updates. It is at a higher level, or so it feels.

Meetings left untouched just failed to show the maps. And right now, that seems to be fixed. But meeting-listings in a half-way state are still stuck, at least for me. People who refresh their whole meeting databases once a week will be in huge trouble, I fear, expecially when they first delete the old content. New content will no longer be accepted until Google services are resumed. What a dilemma!

The settings section of the plugin gives us (webmasters) the impression that we have the choice which product to use for location maps. But nowhere do we see an option for address vetting (or not). So with the two seemingly linked together there is now a perfect storm...

This whole issue might result in the development notion that different paths need to exist in coding for just the website plugin and for its connections to the AA specific usage (that seem to have more and stricter requirements).

It is a testament to the work of the team that the use of what probably started as a simple plugin has gradually become more complex. This will take time and serious work to solve.

I find the plugin indispensable for my website. Really could not do this myself. I am very happy with what it currently has to offer and its ease of use. So I hope a good solution will be found.

Oh well. Let's see where we are tomorrow. I was going to call it a night hours ago and I am getting tired. I guess my HP is telling me to surrender it in good faith.

I cheer the team on from the sideline, trusting in a good solution.

Nicky

PS: To the team: If my take on this is totally wrong, feel free to remove this post :)

[HowardL-md]

Well if you put your Google key in at line 585 there it might solve your problem. I imagine the "hot fix" is to pay that bill for now and figure out what to do before the charges build up again. There must be a ton of sites using this now and the next time they need to update they will have the same issue and be out of commish. Not good.

I would like to have the option to disable the geolocate feature. But like you say there are a lot of cogs and wheels so I don't know if that is possible. Keeping it simple has it's advantages:)

[Area48Webmaster]

@tech2serve I want to start off by saying a big thank you to you and your team for responding so immediately to my original post. I apologize that I was not able to stay involved in the conversation yesterday.

I just created my own Google API key to implement the work around regarding line 585 of the code but I have one question I hope you can help answer. Does the temporary work around open any risk to my area to being charged money? From something I read it may be that mapbox is free?

Best of luck to you and your team, we all greatly appreciate the work you're putting in and understand what a stress these unexpected changes can cause.

[tech2serve]

I just created my own Google API key to implement the work around regarding line 585 of the code but I have one question I hope you can help answer. Does the temporary work around open any risk to my area to being charged money? From something I read it may be that mapbox is free?

Technically, yes. However, we believe that with Google's credit applied, it is unlikely an individual site usage will come anywhere close to exceeding Google's credit. Google does let you set up alerts, and you should. Heck, we provided this for all sites for a long time without a major issue. We're still not sure of why our usage is so much higher than normal and Google doesn't seem to provide good usage analysis tools so we think it isn't worth a lot of our time to try and determine the root issue.

Another step you can take to make your API key more secure is to impose restrictions on it. Assuming your website IP address is static, you should lock down the API key to only accept queries from that address. That will reduce the likelihood of others using your key.

Mapbox geocoding is being worked by one of our volunteers. Mapbox does have a higher limit of usage before they require payments, and we hope that will be a good longer term solution.

[Area48Webmaster]

@tech2serve thank you. What actually trips a "usage" that google keeps track of?

Also, without changing the code on line 585, but just by entering an API key on the Import & Settings page sidebar, and hitting refresh, it seems to be refreshing now without issue. The remaining issue is just that on our website the google map does not load correctly. Does this make sense with what you and your team are working on? If so we can live with that for now!

[tech2serve]

For those who are without websites right now, I believe the API key is turned back on and some additional alerts have been set for the billing. If it gets anywhere close to charging, I'll have to turn the key off again.

At this time, Code for Recovery is not intending for this to be a long term approach. We are working quickly to make it so the API key entered by the user for mapping is also used for geocoding. In the coming days, we'll be testing geocoding through Mapbox. Frankly, we are not certain how well Mapbox geocoding works, nor how well it will integrate with the Meeting Guide. This approach shifts responsibility for managing API keys for a particular website to the group running with that website. This is probably more in tune with 7th Trad anyway, but it will likely involve some growing pains.

Long term, or if the Mapbox geocoding doesn't pan out, we have a couple of other solutions we'll explore, including possibly asking for contributions. Even managing a process like this takes volunteer hours (hint, hint) so it is personally my least favorite option.

What you can do to help right now to help is to avoid meeting list imports as much as possible. Only delete data if absolutely required. Please use the meeting admin page as the tool for entering and changing meetings. It is my belief that large recurring imports of meetings are one of the causes for the significant jump in geocodes. The other cause is likely the newly added support for approximate location tracking (used for the requested online meeting features). It is possible that as sites get the newly tracked data in their databases, the number of geocodes will drop off. But, we just don't know for sure, and we can't afford another bill from Google.

Thanks for hanging with us, and we do appreciate that so many groups (700+ according to Wordpress.org) are using TSML to help manage their meeting lists and provide recovery information to those of us who need these programs. I believe we'll get this resolved and make it even better for the future.

[tech2serve]

@tech2serve thank you. What actually trips a "usage" that google keeps track of?

Also, without changing the code on line 585, but just by entering an API key on the Import & Settings page sidebar, and hitting refresh, it seems to be refreshing now without issue. The remaining issue is just that on our website the google map does not load correctly. Does this make sense with what you and your team are working on? If so we can live with that for now!

Google charges for each geocodes and map display. A couple years ago, mapping was shifted to use either a Mapbox key or Google if users wanted to set the latter up (which is a pain, and requires a credit card). We kept geocoding under a key we provided because we didn't know of any other good option. That is the one we're now having troubles with. Every geocode (entering an address either through the meeting edit page or by importing data) is billed against us.

See my comment above. That may explain why it appears you're seeing oddities with your site right now. Hopefully they'll stabilize over the next couple of hours, but I have seen the error rate go back up the last few minutes. So, now I'm not sure it is on.

I believe the workaround of adding your own key to line 585 should give you the best control at this time.

[OSAwebsite]

I have just been able to update my half-state meeting - phew! Whatever you did, it now works and I also see the maps in the group details pop up again. I have no Google Maps key, I am using a Mapbox key.

If that starts to fail again, might I repeat my earlier request of telling us exactly what Google API would be needed. I asked someone who works with API technology on a daily basis and he told me that we should get directions where to go to get the correct API and restrict it to our own website. I am not doing that at this time, because all seems to be well again.

About your 'hint' about volunteer help: which technology is needed specifically to be able to contribute in terms of working on the code?

I can help think about architecture and specifications from my background as IT database builder, but my knowledge about coding is vertically into a platform that is not used for plugins. Just reach out to me privately if you are interested in that.

Is Google open to you to see what caused the spikes? I mean, they have Analytics about everything, should it not be possible to get insight into what caused the spikes? It probably requires nagging their Sales people on a daily basis but in every other business we, as customers, are entitled to know what we are paying for...

Thanks for the hard work to get this fixed!

[OSAwebsite]

Oh and before I forget: my map on the Details page now shows the 'Mapbox' credit like before insteads of the Google Maps logo... Yey!

[tech2serve]

I have just been able to update my half-state meeting - phew! Whatever you did, it now works and I also see the maps in the group details pop up again. I have no Google Maps key, I am using a Mapbox key.

If that starts to fail again, might I repeat my earlier request of telling us exactly what Google API would be needed. I asked someone who works with API technology on a daily basis and he told me that we should get directions where to go to get the correct API and restrict it to our own website. I am not doing that at this time, because all seems to be well again.

See my comment above about getting the key back on, but this is short term. Above, I've described the steps we're taking.

The person you asked is correct. One of our members has agreed to try and put together a guide. In short, one needs to create a Google billing account and then create an API key (credential) that uses the geocoding and mapping APIs. And finally, restrict it to their website.

Alternatively, we're hopeful the code being put together for Mapbox geocoding will suffice, and then users will have the option of the two services for both geocoding and mapping.

About your 'hint' about volunteer help: which technology is needed specifically to be able to contribute in terms of working on the code?

My hint wasn't about coding (although if someone knows PHP and javascript, we'd welcome more developers). Instead, I was referring to the skills to rename and then maintain a 501c3 structure so we can accept donations and pay for the service in that manner. Frankly, not my first choice, but it is an option and has merit because it makes things easier for the end users. One was created to get the original effort (Meeting Guide) up and running, but we formally have it for Code for Recovery. So, if someone has experience and is willing to take that one, we'd discuss it as a group.

I can help think about architecture and specifications from my background as IT database builder, but my knowledge about coding is vertically into a platform that is not used for plugins. Just reach out to me privately if you are interested in that.

Is Google open to you to see what caused the spikes? I mean, they have Analytics about everything, should it not be possible to get insight into what caused the spikes? It probably requires nagging their Sales people on a daily basis but in every other business we, as customers, are entitled to know what we are paying for... I do, but their analytics haven't been particularly useful in this case. Still trying to figure out how to improve that, but not convinced that really helps us get to the necessary long term position.

Thanks for the hard work to get this fixed!

[Grateful-Viv]

I've been reading through this thread and hope for some clarification.

First, we have none of the issues described in this thread.

We are using MapBox for the TSML plugin. We are using Google Maps for our event plugin and do have a Google Maps API.

Should I go ahead and replace the key found on line 585 of the /wp-content/plugins/12-step-meeting-list/includes/functions.php with our key? And that's it? What happens with a new update to the plugin? Does this then get overwritten?

[tech2serve]

@Grateful-Viv,

You may not have noticed the issue if you didn't do anything with your site after about 0830 (central) this morning.

If you're not having troubles, I wouldn't change anything. We're testing, and then will announce, and then release the change. Because I got the API key back on, I'm willing to take a little more time and try and get this right.

[Area48Webmaster]

@tech2serve with your API key back on is there a way to enter your API key back into line 585? We have no problem manually updating any changes for the time being and not refreshing the import data after the initial update to get all the info back. My area is hesitant about adding billing information for the API key I created. If not just let me know! Thanks.

[HowardL-md]

If we had a google API key would it be helpful for us to use that at line 585 to reduce your google usage ?

That may serve as a long term fix - for each TSML user to supply their own google API key for the geolocation. You could make an input for this just like you have for the Google Maps API Key. Then you wouldn't have to go through all the hubub of collecting donations.

Also , do you have any idea how many requests were sent to google before they start charging? I imagine its several thousand given the number of updates that all the users have made over the time since you have gone live with the plugin. Thanks

[tech2serve]

If we had a google API key would it be helpful for us to use that at line 585 to reduce your google usage ?

Yes.

That may serve as a long term fix - for each TSML user to supply their own google API key for the geolocation. You could make an input for this just like you have for the Google Maps API Key. Then you wouldn't have to go through all the hubub of collecting donations. We agree. The problem is that setting up a Google API key can be daunting for many people, and it it requires a credit card.

That is also why we're trying to get geocoding using Mapbox working.

Also , do you have any idea how many requests were sent to google before they start charging? I imagine its several thousand given the number of updates that all the users have made over the time since you have gone live with the plugin. I could figure it out, but haven't bothered yet. I think I saw where it is $5 for every 10K, and we get a $200 or $250 credit. So, a lot.

But, with over 700 sites and a number of people who reimport their meeting set frequently(?) or at least from time-to-time.

Thanks

[HowardL-md]

Ok so assuming it's a $200 credit: $200 / $5 = 40 chunks of $5 (co5) 40 co5 * 10,000 calls = 400,000 calls to google to use up your credit.

And if the tsml key has gone over that by $200+- that's 800,000 total.

I didn't realize that that all the meetings in our json went through the google mill each time I refreshed which has been once a week for the last 6mo. There have been so many meeting changes every week, and we only have 150! Sorry about that...

The stupid thing is that we don't really use the tsml, it was just there to try as an alternative. We've have our own page for eons which generates off of our json file, we update that. So the best thing for us to do is uninstall. Otherwise I would get my own google key (whilst holding my nose:).

But hi-5 to you guys for all the time and effort you have spent on this very useful tool for recovery.

[tech2serve]

Ok so assuming it's a $200 credit: $200 / $5 = 40 chunks of $5 (co5) 40 co5 * 10,000 calls = 400,000 calls to google to use up your credit.

And if the tsml key has gone over that by $200+- that's 800,000 total.

I didn't realize that that all the meetings in our json went through the google mill each time I refreshed which has been once a week for the last 6mo. There have been so many meeting changes every week, and we only have 150! Sorry about that...

Actually, they don't all go "through the google mill", or at least they shouldn't. The only ones that should go through are those for which your site doesn't have a validated location already in the database. That is part of what we're looking at.

The stupid thing is that we don't really use the tsml, it was just there to try as an alternative. We've have our own page for eons which generates off of our json file, we update that. So the best thing for us to do is uninstall. Otherwise I would get my own google key (whilst holding my nose:).

The plugin exists for people to use; to simplify meeting management and provide a consistent experience for users while feeding data into Meeting Guide. So feel free to use it if it works for you and your users :-)

But hi-5 to you guys for all the time and effort you have spent on this very useful tool for recovery.

Thanks.

At this moment in time, I'm leaning more towards there being an issue with someone's site (or possibly a bug we introduced, but I don't believe we touched the geocoding code) that is causing a bunch of malformed requests to be sent to Google. I'm thinking this because the error rate is >70%, and that started about two weeks ago. Although I'm not sure that is the only issue.

[OSAwebsite]

Instead of guessing, ask Google for a breakdown of the bill. I am sure they have that. Probably by ip-address and from there to the related website if there is indeed a spike caused by one or more massive imports. --

- Yours in service, Nicky D Ontario South Website Coordinator al-anon.alateen.on.ca

This too will pass...

[HowardL-md]

Is there a chance that your key has been hijacked from the source and being used by some other persons?

[SIA-WEBSUP]

Hey Tim, Josh and everyone else contributing to this project,

First off... thank you from the bottom of my heart for all the time and effort you folks have put into this plug-in. It is truly a wonderful resource for so many in the recovery community. You guys deserve lots of props and no grief.

I ran into this issue on Friday night, and had to restore our website from an old backup. Note to self... backup website more often!!

I did try using our own Google Maps API Key in line 585, but ran into a different set error messages. Since I don't wan to the website to be down while I figure this out, I'll have to wait until after hours to figure out using our own API key is working.

When you paid the bill (thank you for that), I was able to re-import our meeting list into the plugin, which I was doing fairly often, ignorant to the fact that this was somehow using up some Google API credit somewhere. Sorry about that.

For now, I will try to avoid any full scale re-imports until I have a good solution to this situation. Ideally, I'll be able to figure out how to use our own API key for the geo-coding as well as the mapping.

With all that said, I want to understand if and why I should be avoiding re-importing our JSON feed into the plug-in. We keep our meeting list as a separate DB which we then build a JSON feed for consumption by the plug-in and the meeting guide app. Previous to this issue, I was perfectly comfortable re-importing the JSON feed ANY time there was an update to our external meeting list DB.

Was this causing every single address to go through the geo-coding on re-import?

Or only the addresses that hadn't gone through the geo-coding already?

Because from the error messages that I saw when the credit had been used up, it seemed like every single entry being imported was causing an error message. As a result, the re-import resulted in NO entries in the plug-in's meeting list DB.

Could we possibly be the culprit who is sending malformed addresses through?? I don't believe so, but how would I find out? Here is our JSON feed: https://suffolkny-aa.org/meeting-guide.php

I can't imagine that we are the only intergroup that would prefer to keep our DB separate from the plug-in and importing the list when necessary.

Any info would be greatly appreciated.

[OSAwebsite]

Hi SIA-WEBSUP,

I want to chime in about the separate meeting database remark you made. I too have that setup, for several reasons. First of all: it is easier to perform searches, sorts and selections in my own database. Second: if ever something goes wrong with the plugin, I still have access to my date. Third: I create district specific and other meeting list PDFs that I cannot create through the website. And there are some more reasons.

However: I do not often do a full or partial refresh, unless there is a large batch that would just take too long to process manually.

I have found that the main loss of time I experience is in opening of the meeting plugin list, then finding the meeting I need to edit and then go to the details page to make my edits. Depending on the day this can take over 5 minutes, while staring at Wordpress reloading. I just don't have the time nor the patience for that.

I have found a workaround that helps me speed this up and that is the main reason I edit my meetings online, for the most part.

I have a direct URL link in my databse, dynamically calculated per meeting, that allows me to jump right to the relevant Details page. This year, most of the changes have to do with Zoom details, so I can simply copy and paste from my database into the plugin page and then hit save. This is a lot quicker than the aforementioned native editing process because it skips a couple of steps. The page I need usually loads within seconds and off I go.

It does mean I make changes in two locations, however, like I said, I can usually copy and paste data into the website instead of re-entering manually.

Also, I have found that deleting the complete list does not always work immediately. Sometimes Wordpress allows me to do it, sometimes now. I have had little success with the import with the box for 'delete meetings' ticked: I end up with duplicate meetings that way. So I don't really trust that setting.

So, having said that: I can totally imagine that when Covid truly exits our life, I will have to do large batches of updates that reflect that meetings are being held face-to-face again. So I hugely appreciate that we even have the import feature.

Now, back to the errors we have had recently due to Google: I would love to have more insight into the issue that caused these troubles. Not sure it is my business though...

Seeing how many analytics are available for everything Google, and looking at common courtesy and accountability when it comes to invoices, I think that Google should be able to show where the root of the problem was. I cannot imagine that Google expects anyone to fork up a lot of money for hundreds of thousands of unanticipated API calls without such clarification and background. It will probably come down to a list of IP addresses (which point to websites) from which the API was called and that means it is probably possible to find out if the cause of the errors is to be found there. Might be a bit of work, but it is possible. It might even be due to bots and hackers, who is to say.

I am always surprised to see robot attacks on the website I maintain. We have no comments, no blogs, no personal details of any kind, yet, we are being targeted by bots. It happens.

Anyway, let's see if the team is able to make geocoding checks via Mapbox work. If it works, there is no need for further discussion. We will all give them a good round of applause and off we go.

If the geocoding vetting via Mapworks does not work, then maybe the software should offer us some choices in future, when all have had time to inspect the issues at hand.

1. If we are an AA program, working with Google APIs probably is a mandatory thing due to the connections between AA Pages that use the plugin and the AA Meeting Guid app. If more moneys are needed as a result, then we can try and cover that through the Seventh Tradition or the GSO budget (GSO is the sole owner of the AA Meeting Guide so I am assuming they have a budget for the people who work on development for that tool)

2. If we are a non-AA 12 Step Program (e.g. I am with Al-Anon), then I would like to be able to choose whether the geocode checks must happen or not. And if so, how.

I am thinking this would be a pretty intuitive process.

Choice 1: editor checks the address themselves If I enter an address for a new meeting, or change one for an existing meeting, the map will either show a location that is correct, a location that is incorrect or not show a location at all. In the two latter scenarios the editor needs to check the location. It could be as simple as a typo. Or an alternative accommodation name. I have had to do that several times when I started out with the plugin for the website I maintain. I typically Google (!) the accommodation and check what comes up against what I have. Sometimes a mere difference in the zip code will do the trick and the map shows. Sometimes the address on file online is different from what a group provided me with. In which case I can choose to contact the group and discuss this. Or I can decide that what is online is in fact the correct accommodation and I simply copy the address shown there. Don't forget that, when looking for a location via a browser search, usually some sort of map comes up, so it is not that hard to check.

Choice 2: have the plugin do the geocode vetting If, as the editor of a non AA program website, I don't feel like having to do all of those checks, I can sign up for the Google API to do the geocode vetting. In which case I should probably be charged a small amount per year, which goes to the plugin people to cover their costs. Maybe the amount can be related to a tiered system of the number of meetings on my pages. Or GSO and, in my case: WSO (Al-Anon World Service Office) agree on a blanket financial arrangement.

Just some thoughts. I had better stop here and wait what the plugin people come back with. For now, everything seems to run smoothly again, yey!

Nicky

[joshc47]

What databases are you all using to feed into TSML?

Josh C

On Sun, Jan 3, 2021 at 4:04 PM SIA-WebSup notifications@github.com wrote:

Hey Tim, Josh and everyone else contributing to this project,

First off... thank you from the bottom of my heart for all the time and effort you folks have put into this plug-in. It is truly a wonderful resource for so many in the recovery community. You guys deserve lots of props and no grief.

I ran into this issue on Friday night, and had to restore our website from an old backup. Note to self... backup website more often!!

I did try using our own Google Maps API Key in line 585, but ran into a different set error messages. Since I don't wan to the website to be down while I figure this out, I'll have to wait until after hours to figure out using our own API key is working.

When you paid the bill (thank you for that), I was able to re-import our meeting list into the plugin, which I was doing fairly often, ignorant to the fact that this was somehow using up some Google API credit somewhere. Sorry about that.

For now, I will try to avoid any full scale re-imports until I have a good solution to this situation. Ideally, I'll be able to figure out how to use our own API key for the geo-coding as well as the mapping.

With all that said, I want to understand if and why I should be avoiding re-importing our JSON feed into the plug-in. We keep our meeting list as a separate DB which we then build a JSON feed for consumption by the plug-in and the meeting guide app. Previous to this issue, I was perfectly comfortable re-importing the JSON feed ANY time there was an update to our external meeting list DB.

Was this causing every single address to go through the geo-coding on re-import?

Or only the addresses that hadn't gone through the geo-coding already?

Because from the error messages that I saw when the credit had been used up, it seemed like every single entry being imported was causing an error message. As a result, the re-import resulted in NO entries in the plug-in's meeting list DB.

Could we possibly be the culprit who is sending malformed addresses through?? I don't believe so, but how would I find out? Here is our JSON feed: https://suffolkny-aa.org/meeting-guide.php

I can't imagine that we are the only intergroup that would prefer to keep our DB separate from the plug-in and importing the list when necessary.

Any info would be greatly appreciated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/code4recovery/12-step-meeting-list/issues/255#issuecomment-753675919, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASKJOB6B5FDXVRVL3U6EPWLSYDLXVANCNFSM4VP7OJ5Q .

[OSAwebsite]

Hi Josh, I have my records in FileMaker Pro, but the format I use for importing is a straight forward comma separated text file.

[SIA-WEBSUP]

Josh,

Suffolk Intergroup Association (SIA - suffolkny-aa.org) uses a MySql to manage a seperate meeting list DB which we then import into the plug-in via the same JSON feed we generate for use by the meeting guide app.

Here are a few of the reasons why I would like to continue having a separate our DB

1. I just don't like having our meeting info inside the larger WordPress MySql DB
2. Our DB is a proper 'groups' DB, rather than a meetings DB... if I make a change at the group level, I don't want to have to edit X meetings where X can be quite large
3. And the biggest reason is that at some point, we'll need to resurrect our process for creating a printed meeting list, which I believe will be easier to do using our existing DB, rather than exporting the DB as it's housed in the plug-in

It's timely that this issue is coming up, because I was at the beginning stages of re-vamping our existing DB. I've been meaning to improve the schema and improve how updates are made.

Josh, do you a rough idea of how many AA entities that use the plug-in have maintain a separate DB, vs. those that use the plug-in at their master DB. In particular, would you know if NYIntergroup uses the plug-in as their master repository and front-end to their meeting list? I ask about NYIntergroup because I imagine they are one of the larger datasets and they (will eventually return to) using the plugin.