search

gregsadetsky / nycnoise

https://nyc-noise.com
18 stars 1 forks source link

day-map #253

Open gregsadetsky opened 8 months ago

gregsadetsky commented 8 months ago

consider options to show mini "map of the day" showing all events on a map

some rando notes below...

have cache of google maps places -> lat lng (via opengraph tags when fetching those place urls...??) // use api and pass place id...?

[out:json];

(
  way["natural"="coastline"]({{bbox}});
);

out body;
>;
out skel qt;
gregsadetsky commented 2 months ago

some more notes/thoughts here:

protomaps would be very cool - we want subways!! mostly... but also roads, obviously... look into all of this!!

or............. as noted in the above comment, maybe a very high level outline is good enough...? i.e. it's mostly about neighborhoods/subway lines....? and that's it?

litviakk commented 2 months ago

@gregsadetsky drafting up a scrapper but I couldn't see any coords in the metas, I tried without headers and it didn't seem to work either. Maybe my link is not right? Can you please give an example of links you have?

gregsadetsky commented 2 months ago

no yeah the link is good - take a look at the page and search for the lat lng, it's there (not super obviously)

you'll see something like window.APP_INITIALIZATION_STATE=[[[3024.9156916505167,-73.9077934,40.6978544]

the two last numbers at the lng lat :-) ... it seems to hold? want to look at some urls and see if you can find this?

litviakk commented 2 months ago

yep, some urls would be nice!

On Tue, Sep 24, 2024 at 11:23 PM Greg Sadetsky @.***> wrote:

no yeah the link is good - take a look at the page and search for the lat lng, it's there (not super obviously)

you'll see something like window.APP_INITIALIZATION_STATE=[[[3024.9156916505167,-73.9077934,40.6978544]

the two last numbers at the lng lat :-) ... it seems to hold? want to look at some urls and see if you can find this?

— Reply to this email directly, view it on GitHub https://github.com/gregsadetsky/nycnoise/issues/253#issuecomment-2372829118, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADC4I4GDYD2WWKEJXFQWAS3ZYIUD7AVCNFSM6AAAAABEFH3Z2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZSHAZDSMJRHA . You are receiving this because you commented.Message ID: @.***>

gregsadetsky commented 2 months ago

for sure!

https://maps.app.goo.gl/5B6wJmsggeQAQHJf8
https://maps.app.goo.gl/atAmED6wkPLdMoC38
https://maps.app.goo.gl/HypPmXet1HZvFB2KA
https://maps.app.goo.gl/86ZdyDin9rL9NdvA8
https://maps.app.goo.gl/cDJvH7NMvJ1JtoLBA
https://maps.app.goo.gl/kNcqeTHbRzs4ej8r9
https://maps.app.goo.gl/wgWefEqTU4VLEHAC8
https://goo.gl/maps/F7U6DBssH7m4ZBbD7
https://goo.gl/maps/kPbrM8ccdHBvRrQu7
https://maps.app.goo.gl/pFjgWhi7U48J1Cb27
https://maps.app.goo.gl/rV7mzttz1WA9J6dF7
https://maps.app.goo.gl/ZVc2YzAQAawSKB6m8
https://maps.app.goo.gl/VNHBMSLpkg8UJY7D7
https://maps.app.goo.gl/ai55Cqy1xY46jLEi8
https://goo.gl/maps/CJsuNbJiYcogrcie9
https://maps.app.goo.gl/V6xSxhTtUKsWCEQR8
https://goo.gl/maps/zrmoaX6oGikWZKmN9
https://goo.gl/maps/NFGUZYCsZqBhvgEU8
https://goo.gl/maps/8EbgtAhUi2HoCPA26
https://goo.gl/maps/8EbgtAhUi2HoCPA26
https://goo.gl/maps/g2vobB6yucgUdqWo6
https://goo.gl/maps/x3q4Ccc4FV9bhHkq9
https://maps.app.goo.gl/Bn2gVYMFx6Miga3c8
https://goo.gl/maps/RbGMaKKAwJ4TDVbF6
https://goo.gl/maps/HGZBPiMMX9n5EXyD8
https://maps.app.goo.gl/tQc4CBw9RAkft2Uj8
https://goo.gl/maps/atLFaJvMGEEzinFY6
https://maps.app.goo.gl/WTQLQA7h4L9AFjc7A
https://maps.app.goo.gl/TSuXLrwihAicmahq9
https://maps.app.goo.gl/Dtw5cR1c9jpbaTHJ8
https://goo.gl/maps/GSBayW2P4q7XjHok6
https://goo.gl/maps/SkQMxcLovix4nN6X9
https://maps.app.goo.gl/4YUuMmmCZPNURtFE9
https://maps.app.goo.gl/2qmpdD5U4GAuiimBA
https://goo.gl/maps/ZTscF5ufD53FKQ278
https://goo.gl/maps/euAmEEYkcAb2s1338
https://maps.app.goo.gl/j59kfWekKqxoeo8T8
https://maps.app.goo.gl/B8CLrwhgCjpzkDww7
https://goo.gl/maps/66i4iQTAG8swrQ6NA
https://goo.gl/maps/ShP3hzCMqJZ2SiE96
litviakk commented 2 months ago

window I don't see anything with coords, I found only this one:

window.WIZ_global_data = {
"DpimGf": false,
"EP1ykd": [
"/_/*",
"/accounts/*"
],
"FdrFJe": "266071119098114025",
"Im6cmf": "/_/DurableDeepLinkUi",
"LVIXXb": 1,
"LoQv7e": false,
"MT7f9b": [],
"MUE6Ne": "DurableDeepLinkUi",
"QrtxK": "",
"S06Grb": "",
"S6lZl": 111881503,
"TSDtV": "%.@.[[null,[[45459555,null,false,null,null,null,\"Imeoqb\"],[45453720,null,false,null,null,null,\"GkrBt\"]],\"CAMSCx0O5fu4KQqznA0K\"]]]",
"Vvafkd": false,
"cfb2h": "boq_durabledeeplinkserver_20240916.08_p0",
"eNnkwf": "",
"eptZe": "/_/DurableDeepLinkUi/",
"fPDxwd": [
97517168,
97684531
],
"gGcLoe": false,
"iCzhFc": true,
"nQyAE": {
"GkrBt": "false"
},
"p9hQne": "https://www.gstatic.com/_/boq-devplatform/_/r/",
"qwAQke": "DurableDeepLinkUi",
"qymVe": "",
"rtQCxc": 240,
"uoqGRe": "j8YGH",
"w2btAe": "%.@.null,null,\"\",false,null,null,true,false]",
"xn5OId": false,
"y2FhP": "prod",
"yFnxrf": 1884,
"zChJod": "%.@.true]"
};
litviakk commented 2 months ago

no yeah the link is good - take a look at the page and search for the lat lng, it's there (not super obviously)

you'll see something like window.APP_INITIALIZATION_STATE=[[[3024.9156916505167,-73.9077934,40.6978544]

the two last numbers at the lng lat :-) ... it seems to hold? want to look at some urls and see if you can find this?

grep APP_INITIALIZATION_STATE returns me nothing and I don't see anything with coords, I found (grep window) only this one:

window.WIZ_global_data = {
    "DpimGf": false,
    "EP1ykd": [
        "/_/*",
        "/accounts/*"
    ],
    "FdrFJe": "266071119098114025",
    "Im6cmf": "/_/DurableDeepLinkUi",
    "LVIXXb": 1,
    "LoQv7e": false,
    "MT7f9b": [],
    "MUE6Ne": "DurableDeepLinkUi",
    "QrtxK": "",
    "S06Grb": "",
    "S6lZl": 111881503,
    "TSDtV": "%.@.[[null,[[45459555,null,false,null,null,null,\"Imeoqb\"],[45453720,null,false,null,null,null,\"GkrBt\"]],\"CAMSCx0O5fu4KQqznA0K\"]]]",
    "Vvafkd": false,
    "cfb2h": "boq_durabledeeplinkserver_20240916.08_p0",
    "eNnkwf": "",
    "eptZe": "/_/DurableDeepLinkUi/",
    "fPDxwd": [
        97517168,
        97684531
    ],
    "gGcLoe": false,
    "iCzhFc": true,
    "nQyAE": {
        "GkrBt": "false"
    },
    "p9hQne": "https://www.gstatic.com/_/boq-devplatform/_/r/",
    "qwAQke": "DurableDeepLinkUi",
    "qymVe": "",
    "rtQCxc": 240,
    "uoqGRe": "j8YGH",
    "w2btAe": "%.@.null,null,\"\",false,null,null,true,false]",
    "xn5OId": false,
    "y2FhP": "prod",
    "yFnxrf": 1884,
    "zChJod": "%.@.true]"
};
litviakk commented 2 months ago

there is also this one:

window.IJ_values = [
    [
        null,
        null,
        "",
        false,
        null,
        null,
        true,
        false
    ],'0','https:\/\/maps.app.goo.gl\/',
    null,'boq_durabledeeplinkserver_20240916.08_p0','maps.app.goo.gl',
    0.0,'','7BRx9LMK8Ntny5lScaHYBg','yc9Meie4ocwezVwsGJfhMA','DEFAULT','',
    2024.0,'https:\/\/maps.app.goo.gl\/4UtdECEnYra6kJwa8',
    null,'ltr',
    false,'https:\/\/accounts.google.com\/AccountChooser?continue\x3dhttps:\/\/maps.app.goo.gl\/4UtdECEnYra6kJwa8\x26hl\x3den-US','https:\/\/accounts.google.com\/ServiceLogin?hl\x3den-US\x26continue\x3dhttps:\/\/maps.app.goo.gl\/4UtdECEnYra6kJwa8','https:\/\/accounts.google.com\/SignOutOptions?continue\x3dhttps:\/\/maps.app.goo.gl\/4UtdECEnYra6kJwa8',
    false,
    true,'en','en-US','en_US','https:\/\/goto2.corp.google.com\/mdtredirect?data_id_filter\x3dmaps.app.goo.gl\x26system_name\x3ddurable-deep-link-server',
    null,
    null,'https:\/\/myaccount.google.com\/privacypolicy?hl\x3den-US',
    false,
    null,
    false,'https:\/\/www.gstatic.com\/_\/boq-devplatform\/_\/r\/','https:\/\/myaccount.google.com\/termsofservice?hl\x3den-US',
    null,
    false,
    null,
]
gregsadetsky commented 2 months ago

Did you try curl'ing / wget'ing a few different urls? I took a look at a few urls and found that initialization thing there.

litviakk commented 2 months ago

I tried a few, though I tried neither curl nor wget

gregsadetsky commented 2 months ago

Yeah do try wget/curl, since the urls (that I sent before) will redirect you multiple times when seen in a regular browser

litviakk commented 2 months ago

Yeah, I figured there might be some differences between them all, will see what is the difference. Will do tonight / tomorrow when I get back to my laptop

On Wed, Sep 25, 2024 at 9:04 AM Greg Sadetsky @.***> wrote:

Yeah do try wget/curl, since the urls (that I sent before) will redirect you multiple times when seen in a regular browser

— Reply to this email directly, view it on GitHub https://github.com/gregsadetsky/nycnoise/issues/253#issuecomment-2374032376, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADC4I4DMUBKXWATMSIE6DGLZYKYGJAVCNFSM6AAAAABEFH3Z2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZUGAZTEMZXGY . You are receiving this because you commented.Message ID: @.***>

litviakk commented 2 months ago

okay, wget did work, and it returns something meaningful with wget headers and I got everything from your list here However, they noticed me and I started getting bot-checked after 20-sh requests...

gregsadetsky commented 2 months ago

Ah that's great! Yeah for sure you'd need to sleep between requests (maybe even a lot) and there might still be pretty severe rate limiting

But there's no real rush in a way. Even it takes a few days. The venues don't change that much. We just need to geocode this initial bunch of 800ish