[BUG] iter_search and index not changing search page

RandomGgames commented 2 years ago

What I have been testing is this:

from cursepy import CurseClient
client = CurseClient()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
for addon in client.iter_search(game_id=GAME_ID, category_id=CAT_ID, search=search):
    print(addon)

What should be happening is it should (If I'm reading the docs correctly at the bottom of https://cursepy.readthedocs.io/en/latest/basic/collection.html#explaining-search-parameters) list the mods on page0, than page1, than page2, etc until the last page of mods, however what seems to be happening is the mods on page0 are just being listed over and over.

As you can see, each page seems to be listed over and over again (print(addon) was changed to print(addon.name) for the image):

I have also tried changing search.index = 1 but each print gave the same list of mods.

I've also noticed the mods at the top/bottom of the page don't always seem to be consistent in order... See mods listed around "Mouse Tweaks":

Just Enough Items (JEI)
Mouse Tweaks
Controlling

Industrial Foregoing
Mouse Tweaks
Controlling

Shadowfacts' Forgelin
Mouse Tweaks
Controlling

Shadowfacts' Forgelin
Mouse Tweaks
Controlling

Shadowfacts' Forgelin
Mouse Tweaks
Controlling
...

where Mouse Tweaks is listed every time while Just Enough Items is listed only once the entire time and Shadowfacts' Forgelin wasn't listed in the first page.

Arweix commented 2 years ago

Try using this instead of iter_search:

from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)

Arweix commented 2 years ago

More information about the search: https://github.com/Owen-Cochell/cursepy/issues/3

RandomGgames commented 2 years ago

Try using this instead of iter_search:

from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)

Gets me the error: ImportError: cannot import name 'MinecraftWrapper' from 'cursepy'

Full code:

from cursepy import MinecraftWrapper
client = MinecraftWrapper()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
client.search_mods(search)

Am I doing something wrong?

RandomGgames commented 2 years ago

I've also just noticed that if I give specific addon IDs that were not showing up from the iter search, I cannot get their file contents, as if they don't exist...? (Might be for a different ticket?)

The below code works fine for addons 306612 and 60089 which were showing up above, however when I give the project id 225608, it throws an id error.

(There is also a little block of code that I believe does exactly what Ticket #6 is requesting to add. Specifically the section "FINDING LATEST 1.18.2 FILE FOR {ID}")

from cursepy import CurseClient
client = CurseClient()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods
#ADDON_ID = 306612 # Fabric API (Curseforge "Project ID" in "About Project" tab on in top right)
# 306612 = Fabric API, in ITER SEARCH
# 60089 = Mouse Tweaks, in ITER SEARCH
# 225608 = WorldEdit, NOT in ITER SEARCH (Causes error)
ADDON_IDS = [60089, 306612, 225608]

#"""ITER SEARCH"""
#search = client.get_search()
#for addon in client.iter_search(game_id=GAME_ID, category_id=CAT_ID, search=search):
#    print(addon.name)

#print(f'{client.game(432) = }\n')
#print(f'{client.addon(ADDON_ID) = }')
#print(f'{client.addon_files(ADDON_ID) = }')

for ID in ADDON_IDS:
    print(f"LISTING ALL 1.18.2 VERSIONS FOR {ID}")
    for file in client.addon_files(addon_id = ID):
        if "1.18.2" in file.version:
            print(f'{file}')
    print("DONE\n")

    print(f"FINDING LATEST 1.18.2 FILE FOR {ID}")
    most_up_to_date_file = None
    for file in client.addon_files(addon_id = ID):
        if "1.18.2" in file.version:
            if most_up_to_date_file == None:
                most_up_to_date_file = file
            if file.id > most_up_to_date_file.id:
                most_up_to_date_file = file
    print(f'{most_up_to_date_file}')
    print("DONE\n")

    print(f"DOWNLOAD LATEST 1.18.2 FILE FOR {ID}")
    import requests
    with open(most_up_to_date_file.file_name, 'wb') as f:
        print(f.write(requests.get(most_up_to_date_file.download_url).content))
    print("DONE\n")


Traceback (most recent call last):
  File "c:\Users\RandomGg\OneDrive\Documents\Minecraft\CursePY\CursePY Testing.py", line 24, in <module>
    for file in client.addon_files(addon_id = ID):
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\wrapper.py", line 264, in addon_files
    return self.handle(8, addon_id)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 987, in handle
    inst = hand.handle(*args, **kwargs)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 408, in handle
    return super().handle()
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 190, in handle
    data = self.format(raw_data)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\forgesvc.py", line 483, in format
    final.append(SVCFile.low_format(file, id))
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\forgesvc.py", line 554, in low_format
    final.append(base.CurseDependency(depen['id'], depen['addonId'], depen['fileId'], depen['type']))
KeyError: 'id''```

Arweix commented 2 years ago

Try using this instead of iter_search:
from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)
Gets me the error: ImportError: cannot import name 'MinecraftWrapper' from 'cursepy'

Full code:
from cursepy import MinecraftWrapper
client = MinecraftWrapper()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
client.search_mods(search)
Am I doing something wrong?

What version of cursepy do you have?

Arweix commented 2 years ago

I've also just noticed that if I give specific addon IDs that were not showing up from the iter search, I cannot get their file contents, as if they don't exist...? (Might be for a different ticket?)

The below code works fine for addons 306612 and 60089 which were showing up above, however when I give the project id 225608, it throws an id error.

(There is also a little block of code that I believe does exactly what Ticket #6 is requesting to add. Specifically the section "FINDING LATEST 1.18.2 FILE FOR {ID}")

from cursepy import CurseClient
client = CurseClient()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods
#ADDON_ID = 306612 # Fabric API (Curseforge "Project ID" in "About Project" tab on in top right)
# 306612 = Fabric API, in ITER SEARCH
# 60089 = Mouse Tweaks, in ITER SEARCH
# 225608 = WorldEdit, NOT in ITER SEARCH (Causes error)
ADDON_IDS = [60089, 306612, 225608]

#"""ITER SEARCH"""
#search = client.get_search()
#for addon in client.iter_search(game_id=GAME_ID, category_id=CAT_ID, search=search):
#    print(addon.name)

#print(f'{client.game(432) = }\n')
#print(f'{client.addon(ADDON_ID) = }')
#print(f'{client.addon_files(ADDON_ID) = }')

for ID in ADDON_IDS:
    print(f"LISTING ALL 1.18.2 VERSIONS FOR {ID}")
    for file in client.addon_files(addon_id = ID):
        if "1.18.2" in file.version:
            print(f'{file}')
    print("DONE\n")

    print(f"FINDING LATEST 1.18.2 FILE FOR {ID}")
    most_up_to_date_file = None
    for file in client.addon_files(addon_id = ID):
        if "1.18.2" in file.version:
            if most_up_to_date_file == None:
                most_up_to_date_file = file
            if file.id > most_up_to_date_file.id:
                most_up_to_date_file = file
    print(f'{most_up_to_date_file}')
    print("DONE\n")

    print(f"DOWNLOAD LATEST 1.18.2 FILE FOR {ID}")
    import requests
    with open(most_up_to_date_file.file_name, 'wb') as f:
        print(f.write(requests.get(most_up_to_date_file.download_url).content))
    print("DONE\n")

Traceback (most recent call last):
  File "c:\Users\RandomGg\OneDrive\Documents\Minecraft\CursePY\CursePY Testing.py", line 24, in <module>
    for file in client.addon_files(addon_id = ID):
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\wrapper.py", line 264, in addon_files
    return self.handle(8, addon_id)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 987, in handle
    inst = hand.handle(*args, **kwargs)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 408, in handle
    return super().handle()
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\base.py", line 190, in handle
    data = self.format(raw_data)
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\forgesvc.py", line 483, in format
    final.append(SVCFile.low_format(file, id))
  File "C:\Users\RandomGg\AppData\Local\Programs\Python\Python39\lib\site-packages\cursepy\handlers\forgesvc.py", line 554, in low_format
    final.append(base.CurseDependency(depen['id'], depen['addonId'], depen['fileId'], depen['type']))
KeyError: 'id''```

I have the same error - https://github.com/Owen-Cochell/cursepy/issues/5

RandomGgames commented 2 years ago

Try using this instead of iter_search:
from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)
Gets me the error: ImportError: cannot import name 'MinecraftWrapper' from 'cursepy' Full code:
from cursepy import MinecraftWrapper
client = MinecraftWrapper()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
client.search_mods(search)
Am I doing something wrong?
What version of cursepy do you have?

1.2.0

Arweix commented 2 years ago

Try using this instead of iter_search:
from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)
Gets me the error: ImportError: cannot import name 'MinecraftWrapper' from 'cursepy' Full code:
from cursepy import MinecraftWrapper
client = MinecraftWrapper()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
client.search_mods(search)
Am I doing something wrong?
What version of cursepy do you have?
1.2.0

It's very strange

Arweix commented 2 years ago

Try using this instead of iter_search:
from cursepy import MinecraftWrapper

client = MinecraftWrapper()
search = client.get_search()

client.search_mods(search)
Gets me the error: ImportError: cannot import name 'MinecraftWrapper' from 'cursepy'

Full code:
from cursepy import MinecraftWrapper
client = MinecraftWrapper()

GAME_ID = 432 # Minecraft
CAT_ID = 6 # Mods

search = client.get_search()
client.search_mods(search)
Am I doing something wrong?

Try this: from cursepy.wrapper import MinecraftWrapper

RandomGgames commented 2 years ago

Try this: from cursepy.wrapper import MinecraftWrapper

That works, however I still only get one page of results. Still having the original issue.


from cursepy.wrapper import MinecraftWrapper
client = MinecraftWrapper()

search = client.get_search()
for index in range(50):
    search.index = index
    for mod in client.search_mods(search):
        print(mod)```

RandomGgames commented 2 years ago

Did some testing and was messing with the URL https://addons-ecs.forgesvc.net/api/v2/addon/search?categoryId=0&gameId=432&gameVersion=1.18.2&gameVersion=Fabric&index=0&pageSize=25&searchFilter=&sectionId=6&sort=0 and I (think) I know what's happening...

&index=0 doesn't set the page index to 0, it's where in the total mods to start from (as if it were a giant index of mods)... so &index=0&pageSize=3 shows me the first 3 mods from the total list and &index=2&pageSize=3 skips the first two, showing the last mod again and the 2 next ones after!

So to set a page index, you have to do something like

index = pageSize * desiredPageIndex

where index is what goes into the HTML, pageSize is however many mods there are per page, and desiredPageIndex is the page number 1, 2, 3 like as if browsing the curseforge page literally. As an example with a page size of 5, page one would be &index=0&pageSize=5 and page 3 would be &index=15&pageSize=5

OwenCochell commented 2 years ago

Hello! Apologies for my super delayed response, these past few weeks have been crazy for me, but I should be more available from here on out.

First off, I have added the MinecraftWrapper to the init file, so the import error you encountered in the first post should now be resolved. Just to showcase this, you can import the class like so:

from cursepy import MinecraftWrapper

This honestly makes a lot more sense then importing them from the wrapper file, so this is (probably) how future wrappers will be imported and used.

I have also fixed the misunderstanding with the index on the SearchParam. Most of these changes have been made in the docs, which makes it clear that the index is NOT the page, and is instead the index of the addon to start showing results for. I have also added the set_page() and the bump_page() methods to the SearchParam class to make traversing pages easier. set_page() sets the index to the given page, and bump_page() will add the size of the page change to the current index.

Finally, I have fixed the iter_search() method, which did not stop iteration correctly. I believe this should solve all of the problems you outlined. The new version can be installed with pip, and the specific version you want is 1.3.0. If anything is missing or you have a better idea on how to implement something, please let me know! Thanks again for pointing this out.

OwenCochell / cursepy

[BUG] iter_search and index not changing search page #7