feat: implement function to procure facility submission details

NabbeunNabi commented 4 months ago

Resolves issue #234

The documentation that describes what I did is: -Google Places API documentation -Pricing Pricing is only $2.83 per 1000 calls

Flow

I made comments in the code to help follow. But this is the flow of the file:

Use puppeteer to open a headerless browser that will wait for the page to redirect to the desktop version of the link. This will get us a url with the coordinates.
From this url regex is used to extract the coordinates that are in every url
The coordinates are then used to do a google places search using a text query and location bias returning the closest relevant response to the query. This is because the coordinates should be as exact as possible. (We might need to think more if they are in the same building. Currently I am returning all the fields because we can implement our own filters later for other things if we scale like wheelchair accessibility.)
The prior three steps are then wrapped up in a final function that can be used

Reasoning for flow

I tried multiple ways to get Tokyo Midtown Clinic will you will see is what I tested in the testing header. Following are the other things I tried and read about:

Using reverse geocoding to use the coordinates to find the exact place. This wasn't very viable as it would return anything that is in the same building whether or not hospital and we shouldn't implement logic that google is already doing for us to filter. I tested it with different options and it never even returned the desired location.
I used other parts of the places api. Again just using coordinates did not work for the previous reason plus I tried to include the parameter types with hospital in the argument array. But that seemed to prioritize hospitals only and clinics is not a viable argument
I used Nearby Search in the documentation but it is less reliable since there are many things nearby the coordinates
I tried to find the place from text using just the url, but it seems like the algorithm they use does not want to use it's own details.
This had me settle on text query with the body that I wrote in the code included in the POST request

Testing Instructions (for now)

Add a .env with a GOOGLE_MAPS_API_KEY

If you have tsx installed run:

tsx ./utils/submissionDataFromGoogleMaps.ts

OR

In the terminal run:

npx tsc utils/submissionDataFromGoogleMaps.ts

And then change the generated file from .js to .cjs so the next command can run the common js generations and run:

node utils/submissionDataFromGoogleMaps.cjs

This was used to generate the following code twice for Tokyo Midtown Clinic using the google places API.

Try 1

  {
    name: 'places/ChIJ30G_l3iLGGARXEK75Rc-3Ww',
    id: 'ChIJ30G_l3iLGGARXEK75Rc-3Ww',
    types: [ 'doctor', 'health', 'point_of_interest', 'establishment' ],
    nationalPhoneNumber: '03-5413-0080',
    internationalPhoneNumber: '+81 3-5413-0080',
    formattedAddress: 'Japan, 〒107-6206 Tokyo, Minato City, Akasaka, 9-chōme−7−１, Midtown Tower, 6F',
    addressComponents: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object]
    ],
    plusCode: {
      globalCode: '8Q7XMP8J+CG',
      compoundCode: 'MP8J+CG Minato City, Tokyo, Japan'
    },
    location: { latitude: 35.6661201, longitude: 139.7312888 },
    viewport: { low: [Object], high: [Object] },
    rating: 2.7,
    googleMapsUri: 'https://maps.google.com/?cid=7844494398308696668',
    websiteUri: 'https://www.tokyomidtown-mc.jp/',
    regularOpeningHours: { openNow: true, periods: [Array], weekdayDescriptions: [Array] },
    utcOffsetMinutes: 540,
    adrFormatAddress: '<span class="country-name">Japan</span>, <span class="postal-code">〒107-6206</span> <span class="region">Tokyo</span>, <span class="street-addres
s">Minato City, Akasaka, 9-chōme−7−１, Midtown Tower</span>, 6F',
    businessStatus: 'OPERATIONAL',
    userRatingCount: 204,
    iconMaskBaseUri: 'https://maps.gstatic.com/mapfiles/place_api/icons/v2/generic_pinlet',
    iconBackgroundColor: '#7B9EB0',
    displayName: { text: 'Tokyo Midtown Clinic', languageCode: 'en' },
    currentOpeningHours: { openNow: true, periods: [Array], weekdayDescriptions: [Array] },
    shortFormattedAddress: 'Minato City, Akasaka, 9-chōme−7−1 6F Midtown Tower',
    reviews: [ [Object], [Object], [Object], [Object], [Object] ],
    photos: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object]
    ],
    restroom: true,
    accessibilityOptions: {
      wheelchairAccessibleParking: true,
      wheelchairAccessibleEntrance: true,
      wheelchairAccessibleRestroom: true
    }
  }
]

Try 2

[
  {
    name: 'places/ChIJ30G_l3iLGGARXEK75Rc-3Ww',
    id: 'ChIJ30G_l3iLGGARXEK75Rc-3Ww',
    types: [ 'doctor', 'point_of_interest', 'health', 'establishment' ],
    nationalPhoneNumber: '03-5413-0080',
    internationalPhoneNumber: '+81 3-5413-0080',
    formattedAddress: 'Japan, 〒107-6206 Tokyo, Minato City, Akasaka, 9-chōme−7−１, Midtown Tower, 6F',
    addressComponents: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object]
    ],
    plusCode: {
      globalCode: '8Q7XMP8J+CG',
      compoundCode: 'MP8J+CG Minato City, Tokyo, Japan'
    },
    location: { latitude: 35.6661201, longitude: 139.7312888 },
    viewport: { low: [Object], high: [Object] },
    rating: 2.7,
    googleMapsUri: 'https://maps.google.com/?cid=7844494398308696668',
    websiteUri: 'https://www.tokyomidtown-mc.jp/',
    regularOpeningHours: { openNow: true, periods: [Array], weekdayDescriptions: [Array] },
    utcOffsetMinutes: 540,
    adrFormatAddress: '<span class="country-name">Japan</span>, <span class="postal-code">〒107-6206</span> <span class="region">Tokyo</span>, <span class="street-addres
s">Minato City, Akasaka, 9-chōme−7−１, Midtown Tower</span>, 6F',
    businessStatus: 'OPERATIONAL',
    userRatingCount: 204,
    iconMaskBaseUri: 'https://maps.gstatic.com/mapfiles/place_api/icons/v2/generic_pinlet',
    iconBackgroundColor: '#7B9EB0',
    displayName: { text: 'Tokyo Midtown Clinic', languageCode: 'en' },
    currentOpeningHours: { openNow: true, periods: [Array], weekdayDescriptions: [Array] },
    shortFormattedAddress: 'Minato City, Akasaka, 9-chōme−7−1 6F Midtown Tower',
    reviews: [ [Object], [Object], [Object], [Object], [Object] ],
    photos: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object]
    ],
    restroom: true,
    accessibilityOptions: {
      wheelchairAccessibleParking: true,
      wheelchairAccessibleEntrance: true,
      wheelchairAccessibleRestroom: true
    }
  }
]

This is the frontend with the autofill data! It has the correct place. For each of these URLS

socket-security[bot] commented 4 months ago

New and removed dependencies detected. Learn more about Socket for GitHub ↗︎

Package	New capabilities	Transitives	Size	Publisher
npm/@types/googlemaps@3.43.3	None	`0`	232 kB	types

View full report↗︎

theyokohamalife commented 3 months ago

This looks like a really good start! My only concern is about the cost. How can we prevent being charged from fake submissions, but still pre-fill the submission fields to satisfy the submission input? Let's discuss the pros and cons with doing this on the backend or frontend if we decide to implement a button for moderators to click to fetch the place data.

NabbeunNabi commented 3 months ago

This looks like a really good start! My only concern is about the cost. How can we prevent being charged from fake submissions, but still pre-fill the submission fields to satisfy the submission input? Let's discuss the pros and cons with doing this on the backend or frontend if we decide to implement a button for moderators to click to fetch the place data.

Based on the discussions I have had with @ermish . As long as the Regex checks are working it will at least be a valid google maps url that is a location due to the redirection and the finding of the coordinates and making sure it is a google maps URL. And at this point it would be easiest to autofill on submission. If we get to the point where people start trying to submit fake URLS with bots we can implement a captcha library to prevent this.

NabbeunNabi commented 3 months ago

This looks like a really good start! My only concern is about the cost. How can we prevent being charged from fake submissions, but still pre-fill the submission fields to satisfy the submission input? Let's discuss the pros and cons with doing this on the backend or frontend if we decide to implement a button for moderators to click to fetch the place data.

Based on the discussions I have had with @ermish . As long as the Regex checks are working it will at least be a valid google maps url that is a location due to the redirection and the finding of the coordinates and making sure it is a google maps URL. And at this point it would be easiest to autofill on submission. If we get to the point where people start trying to submit fake URLS with bots we can implement a captcha library to prevent this.

I actually refactored. I think overall it would be better as a button on the moderation panel. It does take some time. So we want the submission created as quickly as possible. The moderators can then autofill the database with the added mutation.

ermish commented 1 month ago

@NabbeunNabi bump in case you missed my review

ermish commented 1 month ago

@NabbeunNabi and I talked about this. Updated user flow:

The backend will have a mutation for updating the location and triggering the google search.
the front end will have a button on the moderator page to manually trigger the mutation
the front end can update the button to say something like "rerun" when the data already exists. (it can be rerun, but want to call out that it has already been run)

We don't want to auto-run because not all submissions will need it plus it will prevent costs on spam submissions

NabbeunNabi commented 2 weeks ago

@ermish I am going to get to this now. Due to our discussion. With it becoming a mutation where the button will exist in the future. Should I still refactor the locations. Or just fix the mutation name with some comments that help people understand the logic?

ermish commented 1 week ago

@ermish I am going to get to this now. Due to our discussion. With it becoming a mutation where the button will exist in the future. Should I still refactor the locations. Or just fix the mutation name with some comments that help people understand the logic?

if I understand, yes, it should still be moved into the service and an updated mutation

ourjapanlife / findadoc-server