Searching for apps - Githubissues

baltpeter commented 3 months ago

Just as in https://github.com/baltpeter/parse-play/issues/9, we also need a search endpoint here.

baltpeter commented 3 months ago

As usual with Apple, there are quite a few options as for which endpoint to use. Let's go through the ones I am aware of.

baltpeter commented 3 months ago

First of all, as I already mentioned, there is the iTunes Search API, which is already well documented by Apple themselves and actually meant for public consumption.

Example:

https://itunes.apple.com/search?term=facebook&country=DE&media=software&lang=en_us&version=2

That returns (abbreviated):

{
    "resultCount": 50,
    "results": [
        {
            "supportedDevices": [
                "iPhone5s-iPhone5s",
                "iPadAir-iPadAir",
                "iPadAirCellular-iPadAirCellular",
                "iPadMiniRetina-iPadMiniRetina",
                "iPadMiniRetinaCellular-iPadMiniRetinaCellular",
                "iPhone6-iPhone6",
                "iPhone6Plus-iPhone6Plus",
                "iPadAir2-iPadAir2",
                "iPadAir2Cellular-iPadAir2Cellular",
                "iPadMini3-iPadMini3",
                "iPadMini3Cellular-iPadMini3Cellular",
                "iPodTouchSixthGen-iPodTouchSixthGen",
                "iPhone6s-iPhone6s",
                "iPhone6sPlus-iPhone6sPlus",
                "iPadMini4-iPadMini4",
                "iPadMini4Cellular-iPadMini4Cellular",
                "iPadPro-iPadPro",
                "iPadProCellular-iPadProCellular",
                "iPadPro97-iPadPro97",
                "iPadPro97Cellular-iPadPro97Cellular",
                "iPhoneSE-iPhoneSE",
                "iPhone7-iPhone7",
                "iPhone7Plus-iPhone7Plus",
                "iPad611-iPad611",
                "iPad612-iPad612",
                "iPad71-iPad71",
                "iPad72-iPad72",
                "iPad73-iPad73",
                "iPad74-iPad74",
                "iPhone8-iPhone8",
                "iPhone8Plus-iPhone8Plus",
                "iPhoneX-iPhoneX",
                "iPad75-iPad75",
                "iPad76-iPad76",
                "iPhoneXS-iPhoneXS",
                "iPhoneXSMax-iPhoneXSMax",
                "iPhoneXR-iPhoneXR",
                "iPad812-iPad812",
                "iPad834-iPad834",
                "iPad856-iPad856",
                "iPad878-iPad878",
                "iPadMini5-iPadMini5",
                "iPadMini5Cellular-iPadMini5Cellular",
                "iPadAir3-iPadAir3",
                "iPadAir3Cellular-iPadAir3Cellular",
                "iPodTouchSeventhGen-iPodTouchSeventhGen",
                "iPhone11-iPhone11",
                "iPhone11Pro-iPhone11Pro",
                "iPadSeventhGen-iPadSeventhGen",
                "iPadSeventhGenCellular-iPadSeventhGenCellular",
                "iPhone11ProMax-iPhone11ProMax",
                "iPhoneSESecondGen-iPhoneSESecondGen",
                "iPadProSecondGen-iPadProSecondGen",
                "iPadProSecondGenCellular-iPadProSecondGenCellular",
                "iPadProFourthGen-iPadProFourthGen",
                "iPadProFourthGenCellular-iPadProFourthGenCellular",
                "iPhone12Mini-iPhone12Mini",
                "iPhone12-iPhone12",
                "iPhone12Pro-iPhone12Pro",
                "iPhone12ProMax-iPhone12ProMax",
                "iPadAir4-iPadAir4",
                "iPadAir4Cellular-iPadAir4Cellular",
                "iPadEighthGen-iPadEighthGen",
                "iPadEighthGenCellular-iPadEighthGenCellular",
                "iPadProThirdGen-iPadProThirdGen",
                "iPadProThirdGenCellular-iPadProThirdGenCellular",
                "iPadProFifthGen-iPadProFifthGen",
                "iPadProFifthGenCellular-iPadProFifthGenCellular",
                "iPhone13Pro-iPhone13Pro",
                "iPhone13ProMax-iPhone13ProMax",
                "iPhone13Mini-iPhone13Mini",
                "iPhone13-iPhone13",
                "iPadMiniSixthGen-iPadMiniSixthGen",
                "iPadMiniSixthGenCellular-iPadMiniSixthGenCellular",
                "iPadNinthGen-iPadNinthGen",
                "iPadNinthGenCellular-iPadNinthGenCellular",
                "iPhoneSEThirdGen-iPhoneSEThirdGen",
                "iPadAirFifthGen-iPadAirFifthGen",
                "iPadAirFifthGenCellular-iPadAirFifthGenCellular",
                "iPhone14-iPhone14",
                "iPhone14Plus-iPhone14Plus",
                "iPhone14Pro-iPhone14Pro",
                "iPhone14ProMax-iPhone14ProMax",
                "iPadTenthGen-iPadTenthGen",
                "iPadTenthGenCellular-iPadTenthGenCellular",
                "iPadPro11FourthGen-iPadPro11FourthGen",
                "iPadPro11FourthGenCellular-iPadPro11FourthGenCellular",
                "iPadProSixthGen-iPadProSixthGen",
                "iPadProSixthGenCellular-iPadProSixthGenCellular",
                "iPhone15-iPhone15",
                "iPhone15Plus-iPhone15Plus",
                "iPhone15Pro-iPhone15Pro",
                "iPhone15ProMax-iPhone15ProMax"
            ],
            "advisories": [
                "Infrequent/Mild Mature/Suggestive Themes",
                "Infrequent/Mild Profanity or Crude Humour",
                "Infrequent/Mild Alcohol, Tobacco, or Drug Use or References",
                "Infrequent/Mild Sexual Content and Nudity"
            ],
            "isGameCenterEnabled": false,
            "features": ["iosUniversal"],
            "screenshotUrls": [
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource221/v4/e4/bd/90/e4bd9024-b230-1d51-d7e3-3b2634a217a2/f254967c-6ef9-4cc6-b4bf-e89cf016337c_1-iOS-5.5-Home.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource211/v4/85/42/a9/8542a90d-7954-e9dd-4e71-e2c7f650f260/c795c58d-ec37-463c-9ab9-ba0b8bc00ddd_2-iOS-5.5-Reels.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource211/v4/88/78/95/88789523-7cbf-65af-f94f-5e11d194bf5e/0130e841-7bad-4f8c-83c9-61ea0bac7d7b_3-iOS-5.5-Stories.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource221/v4/43/d4/d5/43d4d5fa-4b24-2e6d-6da3-bb700d118ceb/9b761f68-4cd1-4c66-b5f2-76de4f58005f_4-iOS-5.5-Groups.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource221/v4/7f/18/c6/7f18c63a-e3e6-ee8a-eb46-6498ed726b9f/68c29824-a52c-4766-8aea-0a70c7a2f86f_5-iOS-5.5-Profile.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource211/v4/1a/5a/1a/1a5a1a76-7315-f75b-7a19-c94b7dfa651a/28ff37e9-d855-4e6e-a8b4-fb2d585bb9ca_6-iOS-5.5-Marketplace.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource221/v4/cd/eb/30/cdeb30f1-a30a-68ff-643e-97bf8de92ba4/691c88f4-f976-46bb-92e1-5f088e8aa47f_7-iOS-5.5-Share.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource211/v4/3d/d6/76/3dd676e3-454b-ce42-4cc1-17adc7ff0ef6/d49c0719-9500-4504-8a4c-3124b914bafa_8-iOS-5.5-Search_with_AI.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource211/v4/c3/c6/6e/c3c66e8f-1167-ce8d-e778-19cc91915f8e/ee650e7b-1eaa-4de6-81a2-0b35dac16446_9-iOS-5.5-Write_with_AI.png/392x696bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/PurpleSource221/v4/59/ff/94/59ff9419-4bd1-6c15-0c70-2d180e4d4598/89fe27f2-ff0a-4a61-a502-73c2bd05da89_10-iOS-5.5-Notifications.png/392x696bb.png"
            ],
            "ipadScreenshotUrls": [
                "https://is1-ssl.mzstatic.com/image/thumb/Purple211/v4/a4/24/09/a424093a-29ec-fe7f-eb01-dee1be5a4e7d/3d69f402-0570-4aa0-b4ae-3b186d3098f3_iPad_Pro_12.9_2nd_Gen-Home.png/576x768bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/Purple211/v4/15/88/04/158804cf-67d3-d322-43ba-64fca479a8c0/e3a393bd-7c74-4e48-9868-1e93adb97c24_iPad_Pro_12.9_2nd_Gen-Watch.png/576x768bb.png",
                "https://is1-ssl.mzstatic.com/image/thumb/Purple211/v4/15/e0/92/15e09217-9d0b-000a-8824-5442ed426528/88d21855-709c-4c4e-b871-f74053b24d6f_iPad_Pro_12.9_2nd_Gen-Self_Profile.png/576x768bb.png"
            ],
            "appletvScreenshotUrls": [],
            "artworkUrl60": "https://is1-ssl.mzstatic.com/image/thumb/Purple221/v4/d1/7b/48/d17b48bc-21c2-440a-e91a-a76898f5dfed/Icon-Production-0-0-1x_U007emarketing-0-7-0-85-220.png/60x60bb.jpg",
            "artworkUrl512": "https://is1-ssl.mzstatic.com/image/thumb/Purple221/v4/d1/7b/48/d17b48bc-21c2-440a-e91a-a76898f5dfed/Icon-Production-0-0-1x_U007emarketing-0-7-0-85-220.png/512x512bb.jpg",
            "artworkUrl100": "https://is1-ssl.mzstatic.com/image/thumb/Purple221/v4/d1/7b/48/d17b48bc-21c2-440a-e91a-a76898f5dfed/Icon-Production-0-0-1x_U007emarketing-0-7-0-85-220.png/100x100bb.jpg",
            "artistViewUrl": "https://apps.apple.com/de/developer/meta-platforms-inc/id284882218?l=en&uo=4",
            "kind": "software",
            "currentVersionReleaseDate": "2024-05-20T12:00:42Z",
            "releaseNotes": "We’ve updated the app to fix some crashes, make features load faster and support iOS 17!",
            "artistId": 284882218,
            "artistName": "Meta Platforms, Inc.",
            "genres": ["Social Networking"],
            "price": 0.0,
            "description": "Explore, connect and share your interests with real people. \n\nWhether you’re looking for a spark of inspiration with reels or want to dive deeper into something you already love with Marketplace or in groups, you can discover ideas, experiences and people that fuel your interests and help you make progress on the things that matter to you on Facebook.\n\nExplore and expand your interests:\n* Shop for affordable and uncommon stuff on Marketplace and take your hobbies to the next level\n* Personalize your Feed to see more of what you like, less of what you don’t\n* Watch reels for quick entertainment that sparks inspiration\n* Discover creators, small businesses and communities who can help you dive deeper into the things you care about\n\nConnect with people and communities:\n* Join groups to learn tips and tricks from real people who’ve been there, done that\n* Catch up with friends, family and influencers through Feed and stories\n\nShare your world:\n* Effortlessly create reels from trending templates, or let your creativity shine with a full suite of editing tools\n* Customize your profile to choose how you show up and who you share your posts with\n* Turn your hobby into a side hustle by becoming a creator or selling things on Marketplace\n* Celebrate everyday, candid moments with stories, which disappear in 24 hours",
            "sellerName": "Meta Platforms, Inc.",
            "trackId": 284882215,
            "trackName": "Facebook",
            "primaryGenreName": "Social Networking",
            "primaryGenreId": 6005,
            "isVppDeviceBasedLicensingEnabled": true,
            "currency": "EUR",
            "genreIds": ["6005"],
            "bundleId": "com.facebook.Facebook",
            "minimumOsVersion": "13.4",
            "averageUserRatingForCurrentVersion": 4.13501000000000029643842935911379754543304443359375,
            "averageUserRating": 4.13501000000000029643842935911379754543304443359375,
            "trackCensoredName": "Facebook",
            "languageCodesISO2A": [
                "AR",
                "HR",
                "CS",
                "DA",
                "NL",
                "EN",
                "FI",
                "FR",
                "DE",
                "EL",
                "HE",
                "HI",
                "HU",
                "ID",
                "IT",
                "JA",
                "KO",
                "MS",
                "NB",
                "PL",
                "PT",
                "RO",
                "RU",
                "ZH",
                "SK",
                "ES",
                "SV",
                "TH",
                "ZH",
                "TR",
                "UK",
                "VI"
            ],
            "fileSizeBytes": "343912448",
            "sellerUrl": "http://www.facebook.com/mobile",
            "formattedPrice": "Free",
            "contentAdvisoryRating": "12+",
            "userRatingCountForCurrentVersion": 592702,
            "trackViewUrl": "https://apps.apple.com/de/app/facebook/id284882215?l=en&uo=4",
            "trackContentRating": "12+",
            "releaseDate": "2019-02-05T08:00:00Z",
            "version": "464.0.0",
            "wrapperType": "software",
            "userRatingCount": 592702
        },
        // …
    ]
}

baltpeter commented 3 months ago

I don't have a jailbroken iPhone ready and the App Store unfortunately uses cert pinning. I really don't feel like going through that effort right now.

I'll try iTunes on Windows instead.

baltpeter commented 3 months ago

Here's what iTunes on Windows does:

And the old x-apple-store-front trick (#1) trick also works. If I set X-Apple-Store-Front: 143443,30, I get a nice JSON instead of the annoying HTML with a <script> tag that holds the data.

However, this endpoint is still quite annoying.

For one, it fetches the data for all media types (?) at once: iPhone apps, iPad apps, Apple Watch apps, iMessage apps (that's a thing? o.o), books, audiobooks, albums, songs, podcasts, podcast episodes, music videos, TV episodes, TV series

That's a lot of wasted data.

Additionally, the response only contains the metadata for a random-looking unordered assortment of entries:

For the rest, you only get "bubbles" with the IDs:

iTunes then fetches the metadata separately in batches:

baltpeter commented 3 months ago

As I also mentioned before, there is also an internal API of App Store Marketing Tools:

https://tools.applemediaservices.com/api/apple-media/apps/US/search.json?types=apps&term=facebook&limit=25&l=en-US&platform=iphone&additionalPlatforms=iphone,mac,appletv,ipad,watch,web

The big advantage here is that this returns the same format as the get app details endpoint we're using:

That makes me gravitate quite heavily towards this endpoint.

baltpeter commented 3 months ago

The disadvantage is that Apple seems to have locked down the endpoint a bit since I last looked at it. It is now quite finicky with what it accepts. For example, using anything other than limit=25 will result in a 403 (even if I specify a lower limit).

baltpeter commented 3 months ago

As I have also alluded to in the linked issue, this API also lives at https://amp-api-edge.apps.apple.com/v1/catalog/us/search, which as far as I understand it should be the main endpoint in fact (the applemediaservices.com one is just a proxy). For that one, you need a token as I explained in the other issue and so my guess is that the applemediaservices.com just has a very limited token.

Unfortunately, the token from the App Store web page (which we are using for the app details endpoint) doesn't appear to work with the search endpoint—I always get a 403…

baltpeter commented 3 months ago

OK. Well, for the time being, I think we should just use the applemediaservices.com endpoint. The consistency between our functions seems like the strongest argument to me. And if we later find a better way to access the same endpoint, we can always switch.

kkthxbye-code commented 3 months ago

I don't have a jailbroken iPhone ready and the App Store unfortunately uses cert pinning. I really don't feel like going through that effort right now.

Not sure if useful, but here's a search request from an iPhone I did recently (truncated headers). I guess the interesting part is amp-api-search-edge.apps.apple.com - however like you discovered, it doesn't seem possible to use an apple.com token. I opted to use the official search API instead, though it is heavily rate limited.

:method: GET
:scheme: https
:path: /v1/catalog/dk/search?additionalPlatforms=appletv%2Cipad%2Cmac%2Cwatch&associate%5Bapps%5D=app-events&bubble%5Bsearch%5D=apps%2Cdevelopers%2Cgroupings%2Ceditorial-items%2Capp-bundles%2Cin-apps&extend=customArtwork%2CcustomScreenshotsByType%2CcustomScreenshotsByTypeForAd%2CcustomVideoPreviewsByType%2CcustomVideoPreviewsByTypeForAd%2CeditorialBadgeInfo%2CeditorialVideo%2CmessagesScreenshots%2CminimumOSVersion%2CremoteControllerRequirement%2CrequiredCapabilities%2CsupportsFunCamera&extend%5Bapp-events%5D=description%2CproductArtwork%2CproductVideo&extend%5Beditorial-items%5D=showLabelInSearch&include=apps%2Ctop-apps&include%5Beditorial-items%5D=marketing-items%2Cprimary-content&l=en-GB&limit%5Bads-result%5D=4&meta%5Bmarketing-items%5D=metrics&platform=iphone&term=teststring&with=guidedSearch
:authority: amp-api-search-edge.apps.apple.com
x-apple-tz: 7200
cookie: X-Dsid=21237191337
cookie: xt-src=b
cookie: pldfltcid=598ae...
cookie: vrep=CKPZ14...
cookie: tv-pldfltcid=598ae...
cookie: xp_ab=1#IJtoLUK...
cookie: xt-b-ts-21237191337=170...
cookie: mz_at_ssl-21237191337=AwU...
cookie: mz_at0-21237191337=AwQ...
cookie: wosid-lite=mYe...
cookie: fsas=AAA...
icloud-dsid: 21237191337
x-dsid: 21237197987
x-apple-store-front: 143458-2,29 t:apps3
x-apple-iad-request-data: AAAA...
user-agent: AppStore/3.0 iOS/16.7.4 model/iPhone10,6 hwp/t8015 build/20H240 (6; dt:162) AMS/1
x-apple-iad-env-name: AAAA...
x-apple-client-application: com.apple.AppStore
x-apple-i-timezone: CEST
x-apple-i-client-time: 2024-05-14T11:38:...
x-apple-app-store-client-request-id: 046...
authorization: Bearer ey...
accept-language: en-DK
x-apple-i-md-rinfo: 50...
x-apple-adsid: 001...
accept: */*
accept-encoding: br, gzip, deflate
x-apple-i-md-m: LK...
x-apple-i-locale: en_DK
x-apple-i-md: AAA...

baltpeter commented 2 months ago

Thank you, @kkthxbye-code. That's helpful context!

tweaselORG / parse-tunes

Searching for apps #12