search

ckatzorke / howlongtobeat

A simple api for https://howlongtobeat.com/
Do What The F*ck You Want To Public License
338 stars 45 forks source link

Service now returns 404 #40

Closed maysaleba closed 2 years ago

maysaleba commented 2 years ago

It seems that how long to beat has changed how search requests are made, the service is now returning 404

wgroeneveld commented 2 years ago

Yeah if to open up a dev console you can see they radically altered the searching, FINALLY having a semi-decent JSON-enabled POST on URI https://www.howlongtobeat.com/api/search

Here's a sample request payload:

{
  "searchType": "games",
  "searchTerms": [
    "monkey",
    "island"
  ],
  "searchPage": 1,
  "size": 20,
  "searchOptions": {
    "games": {
      "userId": 0,
      "platform": "",
      "sortCategory": "popular",
      "rangeCategory": "main",
      "rangeTime": {
        "min": 0,
        "max": 0
      },
      "gameplay": {
        "perspective": "",
        "flow": "",
        "genre": ""
      },
      "modifier": ""
    },
    "users": {
      "sortCategory": "postcount"
    },
    "filter": "",
    "sort": 0,
    "randomizer": 0
  }
}

And the response:

{
  "color": "blue",
  "title": "",
  "category": "games",
  "count": 11,
  "pageCurrent": 1,
  "pageTotal": 1,
  "pageSize": 20,
  "data": [
    {
      "count": 11,
      "game_id": 10162,
      "game_name": "The Secret of Monkey Island",
      "game_name_date": 0,
      "game_alias": "The Secret of Monkey Island: Special Edition",
      "game_type": "game",
      "game_image": "250px-The_Secret_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 23327,
      "comp_plus": 26078,
      "comp_100": 26747,
      "comp_all": 24791,
      "comp_main_count": 451,
      "comp_plus_count": 115,
      "comp_100_count": 144,
      "comp_all_count": 710,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 2289,
      "count_speedrun": 16,
      "count_backlog": 2737,
      "count_review": 506,
      "review_score": 82,
      "count_playing": 37,
      "count_retired": 180,
      "profile_dev": "Lucasfilm Games",
      "profile_popular": 229,
      "profile_steam": 32360,
      "profile_platform": "Amiga, Atari ST, FM Towns, Mac, Mobile, PC, PlayStation 3, Sega CD, Xbox 360",
      "release_world": 1990
    },
    {
      "count": 11,
      "game_id": 6136,
      "game_name": "Monkey Island 2: LeChuck's Revenge",
      "game_name_date": 0,
      "game_alias": "Monkey Island 2: Special Edition",
      "game_type": "game",
      "game_image": "250px-LeChucks_Revenge_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 24560,
      "comp_plus": 26503,
      "comp_100": 27770,
      "comp_all": 25661,
      "comp_main_count": 251,
      "comp_plus_count": 93,
      "comp_100_count": 96,
      "comp_all_count": 440,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 1520,
      "count_speedrun": 13,
      "count_backlog": 2567,
      "count_review": 310,
      "review_score": 83,
      "count_playing": 29,
      "count_retired": 105,
      "profile_dev": "LucasArts",
      "profile_popular": 208,
      "profile_steam": 32460,
      "profile_platform": "Amiga, FM Towns, Mac, Mobile, PC, PlayStation 3, Xbox 360",
      "release_world": 1991
    },
    {
      "count": 11,
      "game_id": 9822,
      "game_name": "The Curse of Monkey Island",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "game",
      "game_image": "250px-The_Curse_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 1,
      "comp_lvl_mp": 1,
      "comp_lvl_spd": 1,
      "comp_main": 28217,
      "comp_plus": 33439,
      "comp_100": 34866,
      "comp_all": 29957,
      "comp_main_count": 112,
      "comp_plus_count": 21,
      "comp_100_count": 49,
      "comp_all_count": 182,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 555,
      "count_speedrun": 2,
      "count_backlog": 720,
      "count_review": 175,
      "review_score": 85,
      "count_playing": 20,
      "count_retired": 18,
      "profile_dev": "LucasArts",
      "profile_popular": 114,
      "profile_steam": 730820,
      "profile_platform": "Mac, PC",
      "release_world": 1997
    },
    {
      "count": 11,
      "game_id": 3149,
      "game_name": "Escape from Monkey Island",
      "game_name_date": 0,
      "game_alias": "Flucht von Monkey Island",
      "game_type": "game",
      "game_image": "250px-Escape_from_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 37292,
      "comp_plus": 37295,
      "comp_100": 43483,
      "comp_all": 37628,
      "comp_main_count": 63,
      "comp_plus_count": 12,
      "comp_100_count": 19,
      "comp_all_count": 94,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 296,
      "count_speedrun": 2,
      "count_backlog": 574,
      "count_review": 94,
      "review_score": 69,
      "count_playing": 14,
      "count_retired": 12,
      "profile_dev": "LucasArts",
      "profile_popular": 76,
      "profile_steam": 730830,
      "profile_platform": "Mac, PC, PlayStation 2",
      "release_world": 2000
    },
    {
      "count": 11,
      "game_id": 109863,
      "game_name": "Return to Monkey Island",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "game",
      "game_image": "109863_Return_to_Monkey_Island.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 0,
      "comp_plus": 0,
      "comp_100": 0,
      "comp_all": 0,
      "comp_main_count": 0,
      "comp_plus_count": 0,
      "comp_100_count": 0,
      "comp_all_count": 0,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 0,
      "count_speedrun": 0,
      "count_backlog": 21,
      "count_review": 0,
      "review_score": 0,
      "count_playing": 2,
      "count_retired": 1,
      "profile_dev": "Terrible Toybox",
      "profile_popular": 65,
      "profile_steam": 2060130,
      "profile_platform": "Nintendo Switch, PC",
      "release_world": 2022
    },
    {
      "count": 11,
      "game_id": 9599,
      "game_name": "Tales of Monkey Island: Chapter 5 - Rise of the Pirate God",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "dlc",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 10763,
      "comp_plus": 12710,
      "comp_100": 12716,
      "comp_all": 11696,
      "comp_main_count": 28,
      "comp_plus_count": 9,
      "comp_100_count": 14,
      "comp_all_count": 51,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 337,
      "count_speedrun": 0,
      "count_backlog": 916,
      "count_review": 27,
      "review_score": 72,
      "count_playing": 9,
      "count_retired": 55,
      "profile_dev": "Telltale Games",
      "profile_popular": 45,
      "profile_steam": 31219,
      "profile_platform": "PC, PlayStation 3, Wii",
      "release_world": 2009
    },
    {
      "count": 11,
      "game_id": 9598,
      "game_name": "Tales of Monkey Island: Chapter 4 - The Trial and Execution of Guybrush Threepwood",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "dlc",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 11088,
      "comp_plus": 13454,
      "comp_100": 14241,
      "comp_all": 12222,
      "comp_main_count": 29,
      "comp_plus_count": 7,
      "comp_100_count": 16,
      "comp_all_count": 52,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 346,
      "count_speedrun": 0,
      "count_backlog": 907,
      "count_review": 27,
      "review_score": 71,
      "count_playing": 9,
      "count_retired": 59,
      "profile_dev": "Telltale Games",
      "profile_popular": 41,
      "profile_steam": 31209,
      "profile_platform": "PC, PlayStation 3, Wii",
      "release_world": 2009
    },
    {
      "count": 11,
      "game_id": 9597,
      "game_name": "Tales of Monkey Island: Chapter 3 - Lair of the Leviathan",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "dlc",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 10399,
      "comp_plus": 11757,
      "comp_100": 11765,
      "comp_all": 10880,
      "comp_main_count": 28,
      "comp_plus_count": 8,
      "comp_100_count": 17,
      "comp_all_count": 53,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 376,
      "count_speedrun": 0,
      "count_backlog": 903,
      "count_review": 28,
      "review_score": 73,
      "count_playing": 9,
      "count_retired": 54,
      "profile_dev": "Telltale Games",
      "profile_popular": 41,
      "profile_steam": 31199,
      "profile_platform": "PC, PlayStation 3",
      "release_world": 2009
    },
    {
      "count": 11,
      "game_id": 9596,
      "game_name": "Tales of Monkey Island: Chapter 2 - The Siege of Spinner Cay",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "dlc",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 9360,
      "comp_plus": 9566,
      "comp_100": 10598,
      "comp_all": 9630,
      "comp_main_count": 29,
      "comp_plus_count": 13,
      "comp_100_count": 18,
      "comp_all_count": 60,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 398,
      "count_speedrun": 0,
      "count_backlog": 879,
      "count_review": 30,
      "review_score": 71,
      "count_playing": 9,
      "count_retired": 55,
      "profile_dev": "Telltale Games",
      "profile_popular": 41,
      "profile_steam": 31189,
      "profile_platform": "PC, PlayStation 3",
      "release_world": 2009
    },
    {
      "count": 11,
      "game_id": 9595,
      "game_name": "Tales of Monkey Island: Chapter 1 - Launch of the Screaming Narwhal",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "dlc",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 11040,
      "comp_plus": 11306,
      "comp_100": 12401,
      "comp_all": 11573,
      "comp_main_count": 38,
      "comp_plus_count": 12,
      "comp_100_count": 22,
      "comp_all_count": 72,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 459,
      "count_speedrun": 0,
      "count_backlog": 858,
      "count_review": 36,
      "review_score": 70,
      "count_playing": 9,
      "count_retired": 58,
      "profile_dev": "Telltale Games",
      "profile_popular": 41,
      "profile_steam": 31170,
      "profile_platform": "Mobile, PC, PlayStation 3",
      "release_world": 2009
    },
    {
      "count": 11,
      "game_id": 9594,
      "game_name": "Tales of Monkey Island",
      "game_name_date": 0,
      "game_alias": "",
      "game_type": "game",
      "game_image": "250px-Tales_of_Monkey_Island_artwork.jpg",
      "comp_lvl_combine": 0,
      "comp_lvl_sp": 1,
      "comp_lvl_co": 0,
      "comp_lvl_mp": 0,
      "comp_lvl_spd": 1,
      "comp_main": 54605,
      "comp_plus": 54846,
      "comp_100": 57688,
      "comp_all": 54761,
      "comp_main_count": 70,
      "comp_plus_count": 13,
      "comp_100_count": 34,
      "comp_all_count": 117,
      "invested_co": 0,
      "invested_mp": 0,
      "invested_co_count": 0,
      "invested_mp_count": 0,
      "count_comp": 254,
      "count_speedrun": 1,
      "count_backlog": 517,
      "count_review": 114,
      "review_score": 77,
      "count_playing": 3,
      "count_retired": 13,
      "profile_dev": "Telltale Games",
      "profile_popular": 16,
      "profile_steam": 0,
      "profile_platform": "PC, PlayStation 3, Wii",
      "release_world": 2009
    }
  ],
  "displayModifier": null
}

(I ignored the required POST header params for now)

That means the scraper.ts part in this soure code is finally redundant, and it also means a lot of redesign work will be needed to get this thing working again, I think. Happy to help out where I can, but time constraints keep me from submitting a pull request with everything in.

wgroeneveld commented 2 years ago

Did a bit more probing, less simple than it looks, keep on getting 401, there's a login API as well. I presume the change is related to https://www.howlongtobeat.com/forum/thread/2674/1

joekw commented 2 years ago

Seems to be enough to include this line in your headers:

Referer: https://howlongtobeat.com/

maysaleba commented 2 years ago

Seems to be enough to include this line in your headers:

Referer: https://howlongtobeat.com/

I get Internal Server Error when I try this in postman

joekw commented 2 years ago

You'll need to include in your POST request all the items in the JSON @wgroeneveld posted above.

wgroeneveld commented 2 years ago

You'll need to include in your POST request all the items in the JSON @wgroeneveld posted above.

Forgot that part, lol. Actually, this makes this node module redundant, as the API makes it trivial to use fetch()...

rarDevelopment commented 2 years ago

Yeah making it a POST request with @wgroeneveld's post above and the header of Referer: https://howlongtobeat.com/ that @joekw mentioned. I'm not quite sure why it's a POST request honestly, I guess because it's a "search" instead of a straight up "GET", but a bit odd.

rafal-dziegielewski commented 2 years ago

Hey, does anyone know the time format that's now included in the json response? For Elden Ring the comp_main shows "190598" which on the HLTB website translates to 53 hours. I can't figure out what type of unit that is, tried ticks, miliseconds, seconds, etc. but can't get it right.

maysaleba commented 2 years ago

Hey, does anyone know the time format that's now included in the json response? For Elden Ring the comp_main shows "190598" which on the HLTB website translates to 53 hours. I can't figure out what type of unit that is, tried ticks, miliseconds, seconds, etc. but can't get it right.

Dividing it by 3600 should give you the hour equivalent

rafal-dziegielewski commented 2 years ago

Dividing it by 3600 should give you the hour equivalent

Rockstar! Thanks!

ckatzorke commented 2 years ago

Thank you for the active discussion (and pointing out all the details for the "pseudo"-api) - and sorry for my late response. Since this will be probably the closest we will get to a real Howlongtobeat API, I almost see no reason to update this wrapper anymore (since most of the work was in html parsing). However, I tried to adopt to the new search API and keep the HowLongToBeatService and HowLongToBeatEntry model compatible (I dropped the timelabels: [][] field (is now null) - in case you miss it :-) ).

If you are interested in, a new version 1.6.0 is published. Be aware that the #detail method is not working anymore (but also was not working in the previous version, and I doubt anyone is using it anyway).

Even if this is almost an API, I would prefer the HLTB crew would rather provide a real API, with rate limiting/budget and even authentication and authorization - up to demanding money for professional use cases (it is now even used in XBox Gamepass, as also referenced above). I always felt like scraping them will not give them in any way the kudos they deserve. So, please be kind, do not bomb them with unnecessary requests, tell everyone about the great service you are using underneath...

And, I would also recommend to get rid off this now almost useless wrapper library, and trigger the search endpoint directly...

Cheers, Christian