deepghs / waifuc

Efficient Train Data Collector for Anime Waifu
https://deepghs.github.io/waifuc/
MIT License
290 stars 24 forks source link

PixivSearchSource module usage exception #60

Closed nameless0412 closed 4 months ago

nameless0412 commented 5 months ago

When I use the PixivUserSource and PixivRankingSource modules, everything works perfectly. However, when I use the PixivSearchSource, the script exits directly. Can you please tell me the reason? Thank you very much. Below are the command line interface instructions and a screenshot. PS D:\waifuc> python waifu.py SaveExporter('/data/amiya_pixiv'): 0it [00:00, ?it/s] WARNING:root:Illusts not found in page (offset: 0), skipped: {'error': {'user_message': '', 'message': '{"offset":["offset must be an integer","offset must be no more than 5000"]}', 'reason': '', 'user_message_details': {}}}. | 0/50 [00:00<?, ?it/s] PixivSearchSource('アークナイツ アーミヤ'): 0it [00:01, ?it/s] FirstNSelectAction(50): 0%| | 0/50 [00:01<?, ?it/s] SaveExporter('/data/amiya_pixiv'): 0it [00:01, ?it/s] PS D:\waifuc> Snipaste_2024-06-20_03-17-28

nameless0412 commented 5 months ago

Added, this is the script file I ran: from waifuc.export import SaveExporter from waifuc.source import PixivSearchSource

if name == 'main': s = PixivSearchSource(

'明日方舟 阿米娅'

    'アークナイツ アーミヤ',
    refresh_token='xQ2cfGGXvgwb5X3KjGZBhhMS0Xfe1SEeJ-zCen-XTXTPvDW8',
)
s[:50].export(
    SaveExporter('/data/amiya_pixiv')
)
narugo1992 commented 5 months ago

@nameless0412 I cannot reproduce your error, it is runnable when im using my refresh token

import os

from waifuc.export import SaveExporter
from waifuc.source import PixivSearchSource

if __name__ == '__main__':
    s = PixivSearchSource(
        # '明日方舟 阿米娅'
        'アークナイツ アーミヤ',
        refresh_token=os.environ['PIXIV_REFRESH_TOKEN'],
    )
    s[:50].export(
        SaveExporter('/data/amiya_pixiv')
    )

btw, when i run your code, the error is

pixivpy3.utils.PixivError: [ERROR] auth() failed! check refresh_token.
HTTP 400: {"has_error":true,"errors":{"system":{"message":"Invalid refresh token","code":1508}},"error":"invalid_grant"}

are you sure your token is correct?

nameless0412 commented 5 months ago

@narugo1992 I used the same token, and the script that retrieved images using userID worked, but the script that retrieved images using role names did not work. I confirmed multiple times that the tokens of the two scripts were exactly the same!!! Below I will post the code I used and a screenshot of the running interface

23e9f418139eb04296d66031b11c69b7

D:\waifuc>WaifuUser.py
PixivUserSource(2864095): 50it [01:00,  1.22s/it]it [00:59,  1.06s/it]
FirstNSelectAction(50): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [01:00<00:00,  1.22s/it]
SaveExporter('/data/pixiv_user_misaka_12003'): 50it [01:00,  1.22s/it]

D:\waifuc>waifu.py
SaveExporter('/data/surtr_arknights_pixiv'): 0it [00:00, ?it/s]                    WARNING:root:Illusts not found in page (offset: 0), skipped: {'error': {'user_message': '', 'message': '{"offset":["offset must be an integer","offset must be no more than 5000"]}', 'reason': '', 'user_message_details': {}}}.                                              | 0/10 [00:00<?, ?it/s]
PixivSearchSource('アークナイツ (surtr OR スルト OR 史尔特尔)'): 0it [00:02, ?it/s]
FirstNSelectAction(10):   0%|                                                                                                                                        | 0/10 [00:02<?, ?it/s]
SaveExporter('/data/surtr_arknights_pixiv'): 0it [00:02, ?it/s]

This script is the one named WaifuUser.py above CMD

from waifuc.export import SaveExporter
from waifuc.source import PixivUserSource

if __name__ == '__main__':
    s = PixivUserSource(
        2864095,  # 爬取用户2864095
        refresh_token='xQ2cfGGXvgwb5X3KjGZBMS0Xe1SEJ-zCn-XTXTPvDW8',
    )
    s[:50].export(
        SaveExporter('/data/pixiv_user_misaka_12003')
    )

This script is the one named waifu.py above CMD

from waifuc.action import HeadCountAction, AlignMinSizeAction, CCIPAction
from waifuc.export import SaveExporter
from waifuc.source import PixivSearchSource

if __name__ == '__main__':
    source = PixivSearchSource(
        'アークナイツ (surtr OR スルト OR 史尔特尔)',
        refresh_token='xQ2cfGGXvgwb5X3KjGZBMS0Xe1SEJ-zCn-XTXTPvDW8'
    )
    source.attach(
        # only 1 head,
        HeadCountAction(1),

        # pixiv often have some irrelevant character mixed in
        # so CCIPAction is necessary here to drop these images
        CCIPAction(),

        # if shorter side is over 640, just resize it to 640
        AlignMinSizeAction(640),
    )[:10].export(  # only first 10 images
        # save images (with meta information from danbooru site)
        SaveExporter('/data/surtr_arknights_pixiv')
    )
narugo1992 commented 4 months ago

@nameless0412 which waifuc and pixivpy3 version are u using?

narugo1992 commented 4 months ago

merged in #63

now you can pull the code in main branch, reinstall it (this is VERY IMPORTANT), and try again

narugo1992 commented 4 months ago

closed due to no further feedback