search

mansuf / mangadex-downloader

A command-line tool to download manga from MangaDex, written in Python.
https://mangadex-dl.mansuf.link/
MIT License
472 stars 37 forks source link

sqlite3.OperationalError: no such table: file_info_cbz #110

Closed vonProteus closed 2 months ago

vonProteus commented 3 months ago

What happened ?

i get exeption

What did you expect to happen ?

download as usual

OS version

docker image python:3

App version

mangadex-downloader v2.10.3 (https://github.com/mansuf/mangadex-downloader) Python: 3.12.3 arch: x64 bundled executable: no

Installation origin

Other

Installation origin (other sources)

https://github.com/mansuf/mangadex-downloader/commit/ae3ab36cd2cc4b8ac2cfd241fae5ac604af4093e

Reproducible command

mangadex-downloader --language en --sort-by chapter --save-as "cbz" --delay-requests 1.5 /downloads/download.txt

Additional context

after i checkout to v3.0.0 and i run mangadex-downloader --language en --sort-by chapter --save-as "cbz" --delay-requests 1.5 /downloads/download.txt i get 

[INFO] Checking url = https://mangadex.org/title/#MANGA_URL#
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/#MANGA#"
[INFO] Using cbz format
[INFO] Downloading cover manga #MANGA#
File sizes: 100%|████████████████████████████| 485k/485k [00:00<00:00, 16.1MB/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/__init__.py", line 87, in _main
    download(args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/download.py", line 34, in download
    url(args, args.type)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 187, in __call__
    self.func(self.id, args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 105, in download_manga
    dl_manga(*args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 200, in download
    download_manga(manga, formatted_path)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 136, in download_manga
    m.tracker = get_tracker(save_as, path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/__init__.py", line 162, in get_tracker
    return DownloadTrackerSQLite(fmt, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 81, in __init__
    self.init_write_mode()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 99, in init_write_mode
    self._load()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 388, in _load
    sql_migrate(self.db)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/__init__.py", line 33, in migrate
    migrate_cls.migrate()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 117, in migrate
    volume_values = self._get_values_for_volume_column()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 34, in _get_values_for_volume_column
    fi_cursor.execute(f"SELECT * FROM file_info_{self.get_format()}")
sqlite3.OperationalError: no such table: file_info_cbz

i don't think it is connected to https://github.com/mansuf/mangadex-downloader/issues/108 but i was testing it

i had to set env MANGADEXDL_CONFIG_ENABLED=1 before that i got

[INFO] Checking url = https://mangadex.org/title/#MANGA#
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/#MANGA#"
[ERROR] Config is not enabled. You can enable it by set MANGADEXDL_CONFIG_ENABLED to 1
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/download.py", line 34, in download
    url(args, args.type)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 187, in __call__
    self.func(self.id, args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 105, in download_manga
    dl_manga(*args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 200, in download
    download_manga(manga, formatted_path)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 101, in download_manga
    config.save_as = backup_fmt
    ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/config/config.py", line 308, in __setattr__
    _conf.write(name, val)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/config/config.py", line 235, in write
    raise MangaDexException(
mangadex_downloader.errors.MangaDexException: Config is not enabled. You can enable it by set MANGADEXDL_CONFIG_ENABLED to 1
[INFO] Cleaning up...

here is docker file i run https://github.com/vonProteus/mangadex-downloader-docker/tree/tmp-3.0.0

mansuf commented 3 months ago

About the error

sqlite3.OperationalError: no such table: file_info_cbz

You solved it by enabling env MANGADEXDL_CONFIG_ENABLED=1 ?

As far as i know, i implemented migrations to database, and it should works for the first time

vonProteus commented 3 months ago

no first i get error you have to set env after setting it i get SQLite error i tried it on new manga and existing same result

mansuf commented 3 months ago

Can you send me download.db file from the download directory ? I wanna inspect it, because in my end it works fine. I hope it's just permission error that the database cannot write it like this one https://github.com/mansuf/mangadex-downloader/issues/52

mansuf commented 3 months ago

For now, you might wanna disable download tracking by enabling --no-track option, it will continue download as usual but you won't find download.db file. By enabling --no-track option, the app cannot check the latest chapters.

vonProteus commented 3 months ago

download.db.zip from new manga with 3.0.0 looks empty to me

mansuf commented 3 months ago

Looking up your download.db it seems the migration file 00000_init.py hasn't applied correctly, you can try see it by setting --log-level=DEBUG when downloading manga.

Output

sussyuser@SUSSY-PC:~/md-test$ docker run mangadex-dl:v3.0.0 "tamamo no koi" -s --input-pos 1 -f cbz --verbose
[WARNING] --verbose is deprecated and will be removed in v3.0.0. Use '--log-level=DEBUG' instead
[DEBUG] Config 'stacked_progress_bar_order' (value: ['volumes', 'chapters', 'pages', 'file sizes', 'convert']) is not passed validator test, reason: 'list' object has no attribute 'split'. Falling back to default value
[DEBUG] Loaded config from cli args = {'login_cache': False, 'language': 'en', 'cover': 'original', 'save_as': 'cbz', 'use_chapter_title': False, 'use_compressed_image': False, 'force_https': False, 'path': './{manga.title}', 'filename_chapter': '{chapter.simple_name}{file_ext}', 'filename_volume': 'Vol. {volume}{file_ext}', 'filename_single': 'All chapters{file_ext}', 'dns_over_https': None, 'no_group_name': False, 'sort_by': 'volume', 'no_progress_bar': False, 'http_retries': 5, 'write_tachiyomi_info': False, 'download_mode': 'default', 'use_chapter_cover': False, 'use_volume_cover': False, 'no_track': False, 'volume_cover_language': None, 'stacked_progress_bar_order': 'volumes, chapters, pages, file sizes, convert', 'log_level': 'DEBUG', 'progress_bar_layout': 'default', 'ignore_missing_chapters': False, 'create_no_volume': False, 'create_manga_info': False, 'manga_info_format': 'csv', 'manga_info_filepath': '{download_path}/manga_info.{manga_info_format}'}
[INFO] Checking url = 433e77d2-5a58-48a3-95b8-e3c02f309255
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/Tamamo no Koi"
[INFO] Using cbz format
[INFO] Downloading cover manga Tamamo no Koi
File sizes: 100%|████████████████████████████| 260k/260k [00:00<00:00, 21.1MB/s]
[DEBUG] Applying download tracker database migration 00000_init.py...
[DEBUG] Applying download tracker database migration 00001_add_table_db_info_and_alter_table_file_info.py...
[INFO] Preparing to download...
[INFO] Getting images from chapter 0
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 1
File sizes: 100%|████████████████████████████| 695k/695k [00:00<00:00, 5.33MB/s]
[DEBUG] Reporting {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/2cb2a15b58b96bc72b705849ce15ed8b/1-42b91b72bbce2b2f62c24c4c4d56a84d79bc6952139e19924a7a6acc72ed13eb.png', 'success': True, 'cached': False, 'bytes': 694683, 'duration': 425} to MangaDex network
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 2

I tried installing your docker image to my server and test it and it works without errors.

mansuf commented 3 months ago

Maybe you can try these python script to test it out if migration is really works. Make sure to test it in your docker container.

import sqlite3
from mangadex_downloader.tracker.sql_migrations import _iter_migrate_cls
from mangadex_downloader.tracker.sql_migrations.base import SQLMigration
from mangadex_downloader.tracker.sqlite import DownloadTrackerSQLite
from pathlib import Path

db_path = DownloadTrackerSQLite.get_tracker_path("cbz", Path("./Tamamo no Koi"))
db = sqlite3.connect(db_path)

cls: SQLMigration
for cls, file in _iter_migrate_cls():
    if file != "00000_init.py":
        continue

    print(cls.check_if_migrate_is_possible())
    cls.migrate()
vonProteus commented 3 months ago

first i build docker build --progress=plain -t mangadex-downloader . from https://github.com/vonProteus/mangadex-downloader-docker/commit/43ca28c07dfd9c9a65581551366302f0d67904b0 and when i run docker run -ti -v ./downloads:/downloads/ --rm mangadex-downloader "tamamo no koi" -s --input-pos 1 -f cbz --log-level=DEBUG i get

[DEBUG] Config 'stacked_progress_bar_order' (value: ['volumes', 'chapters', 'pages', 'file sizes', 'convert']) is not passed validator test, reason: 'list' object has no attribute 'split'. Falling back to default value
[DEBUG] Loaded config from cli args = {'login_cache': False, 'language': 'en', 'cover': 'original', 'save_as': 'cbz', 'use_chapter_title': False, 'use_compressed_image': False, 'force_https': False, 'path': './{manga.title}', 'filename_chapter': '{chapter.simple_name}{file_ext}', 'filename_volume': 'Vol. {volume}{file_ext}', 'filename_single': 'All chapters{file_ext}', 'dns_over_https': None, 'no_group_name': False, 'sort_by': 'volume', 'no_progress_bar': False, 'http_retries': 5, 'write_tachiyomi_info': False, 'download_mode': 'default', 'use_chapter_cover': False, 'use_volume_cover': False, 'no_track': False, 'volume_cover_language': None, 'stacked_progress_bar_order': 'volumes, chapters, pages, file sizes, convert', 'log_level': 'DEBUG', 'progress_bar_layout': 'default', 'ignore_missing_chapters': False, 'create_no_volume': False, 'create_manga_info': False, 'manga_info_format': 'csv', 'manga_info_filepath': '{download_path}/manga_info.{manga_info_format}'}
[INFO] Checking url = 433e77d2-5a58-48a3-95b8-e3c02f309255
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/Tamamo no Koi"
[INFO] Using cbz format
[INFO] Downloading cover manga Tamamo no Koi
[INFO] File exist and replace is False, cancelling download...
[DEBUG] Applying download tracker database migration 00001_add_table_db_info_and_alter_table_file_info.py...
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/__init__.py", line 87, in _main
    download(args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/download.py", line 34, in download
    url(args, args.type)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 187, in __call__
    self.func(self.id, args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/cli/url.py", line 105, in download_manga
    dl_manga(*args)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 200, in download
    download_manga(manga, formatted_path)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/main.py", line 136, in download_manga
    m.tracker = get_tracker(save_as, path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/__init__.py", line 162, in get_tracker
    return DownloadTrackerSQLite(fmt, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 81, in __init__
    self.init_write_mode()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 99, in init_write_mode
    self._load()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sqlite.py", line 388, in _load
    sql_migrate(self.db)
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/__init__.py", line 33, in migrate
    migrate_cls.migrate()
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 117, in migrate
    volume_values = self._get_values_for_volume_column()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 34, in _get_values_for_volume_column
    fi_cursor.execute(f"SELECT * FROM file_info_{self.get_format()}")
sqlite3.OperationalError: no such table: file_info_cbz

and in ./downloads Tamamo no Koi.zip

mansuf commented 3 months ago

[DEBUG] Applying download tracker database migration 00001_add_table_db_info_and_alter_table_file_info.py...

See, before this line of output, you should see:

[DEBUG] Applying download tracker database migration 00000_init.py...

And from your output, the 00000_init.py file is missing. You should check migration files under mangadex_downloader/tracker/sql_migrations directory

vonProteus commented 3 months ago

i think its related to v3 somehow and my not perfect docker image i will wait for you image

vonProteus commented 3 months ago

in my image v3

root@077fe700dbb5:/downloads# ls -lah "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sql_migrations/"
total 32K
drwxr-xr-x 3 root root 4.0K Jun  5 15:57 .
drwxr-xr-x 6 root root 4.0K Jun  5 15:57 ..
-rw-r--r-- 1 root root 1.5K Jun  5 15:57 00000_init.py
-rw-r--r-- 1 root root 5.0K Jun  5 15:57 00001_add_table_db_info_and_alter_table_file_info.py
-rw-r--r-- 1 root root  936 Jun  5 15:57 __init__.py
drwxr-xr-x 2 root root 4.0K Jun  5 15:57 __pycache__
-rw-r--r-- 1 root root 3.6K Jun  5 15:57 base.py
mansuf commented 3 months ago

i will wait for you image

I have uploaded the docker image https://hub.docker.com/r/mansuf/mangadex-downloader/tags, can you try it

vonProteus commented 3 months ago

its works fine but this is version 2.x (without urls in cbz) and i think your docker file is bit big i would suggest something more like

FROM python:3.11

COPY . /app
WORKDIR /app

RUN  pip install .[optional]

WORKDIR /downloads/

ENTRYPOINT [ "mangadex-downloader" ]

CMD [ "--help" ]

its work fine on arm64 (where i tested it)

and its make smaler image 

docker image ls
REPOSITORY                     TAG       IMAGE ID       CREATED              SIZE
mangadex-downloader            latest    4577ce5f1828   About a minute ago   1.3GB
mansuf/mangadex-downloader     latest    b865933e4259   2 hours ago          2.99GB

this whole bug is about version 3

mansuf commented 3 months ago

I have to install some tools that depends on the application itself (most notably rust). So yeah that's why the image is so big compared yours.

i would suggest something more like

You might wanna see this CI fails if the Dockerfile would like that

https://github.com/mansuf/mangadex-downloader/actions/runs/9384224877

and it continue until i figured out how to install rust toolchain in the Dockerfile

https://github.com/mansuf/mangadex-downloader/actions https://github.com/mansuf/mangadex-downloader/actions/runs/9388023850

mansuf commented 3 months ago

this whole bug is about version 3

I'm really sorry i didn't notice that.

From your output checking sql_migrations directory https://github.com/mansuf/mangadex-downloader/issues/110#issuecomment-2150432020, you have the 00000_init.py file and yet the migration is not applied tho, why ? I'm really confused

mansuf commented 3 months ago

Can you help me add these script https://github.com/mansuf/mangadex-downloader/issues/110#issuecomment-2148918844 and run it in your docker image ?

I really wanna make sure what is wrong with the migration

You can use same command to download test manga from here https://github.com/mansuf/mangadex-downloader/issues/110#issuecomment-2149956773 and then run the script

docker run -ti -v ./downloads:/downloads/ --rm mangadex-downloader "tamamo no koi" -s --input-pos 1 -f cbz --log-level=DEBUG

# And then run the migration script
vonProteus commented 3 months ago

in container from 3.0.0 branch mangadex-downloader "tamamo no koi" -s --input-pos 1 -f cbz --log-level=DEBUG

[DEBUG] Config 'stacked_progress_bar_order' (value: ['volumes', 'chapters', 'pages', 'file sizes', 'convert']) is not passed validator test, reason: 'list' object has no attribute 'split'. Falling back to default value
[DEBUG] Loaded config from cli args = {'login_cache': False, 'language': 'en', 'cover': 'original', 'save_as': 'cbz', 'use_chapter_title': False, 'use_compressed_image': False, 'force_https': False, 'path': './{manga.title}', 'filename_chapter': '{chapter.simple_name}{file_ext}', 'filename_volume': 'Vol. {volume}{file_ext}', 'filename_single': 'All chapters{file_ext}', 'dns_over_https': None, 'no_group_name': False, 'sort_by': 'volume', 'no_progress_bar': False, 'http_retries': 5, 'write_tachiyomi_info': False, 'download_mode': 'default', 'use_chapter_cover': False, 'use_volume_cover': False, 'no_track': False, 'volume_cover_language': None, 'stacked_progress_bar_order': 'volumes, chapters, pages, file sizes, convert', 'log_level': 'DEBUG', 'progress_bar_layout': 'default', 'ignore_missing_chapters': False, 'create_no_volume': False, 'create_manga_info': False, 'manga_info_format': 'csv', 'manga_info_filepath': '{download_path}/manga_info.{manga_info_format}'}
[INFO] Checking url = 433e77d2-5a58-48a3-95b8-e3c02f309255
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/Tamamo no Koi"
[INFO] Using cbz format
[INFO] Downloading cover manga Tamamo no Koi
File sizes: 100%|████████████████████████████| 260k/260k [00:00<00:00, 2.37MB/s]
[DEBUG] Applying download tracker database migration 00001_add_table_db_info_and_alter_table_file_info.py...
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/cli/__init__.py", line 87, in _main
    download(args)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/cli/download.py", line 34, in download
    url(args, args.type)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/cli/url.py", line 187, in __call__
    self.func(self.id, args)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/cli/url.py", line 105, in download_manga
    dl_manga(*args)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/main.py", line 200, in download
    download_manga(manga, formatted_path)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/main.py", line 136, in download_manga
    m.tracker = get_tracker(save_as, path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/__init__.py", line 162, in get_tracker
    return DownloadTrackerSQLite(fmt, path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py", line 81, in __init__
    self.init_write_mode()
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py", line 99, in init_write_mode
    self._load()
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py", line 388, in _load
    sql_migrate(self.db)
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sql_migrations/__init__.py", line 33, in migrate
    migrate_cls.migrate()
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 117, in migrate
    volume_values = self._get_values_for_volume_column()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py", line 34, in _get_values_for_volume_column
    fi_cursor.execute(f"SELECT * FROM file_info_{self.get_format()}")
sqlite3.OperationalError: no such table: file_info_cbz

cat script.py

import sqlite3
from mangadex_downloader.tracker.sql_migrations import _iter_migrate_cls
from mangadex_downloader.tracker.sql_migrations.base import SQLMigration
from mangadex_downloader.tracker.sqlite import DownloadTrackerSQLite

db_path = DownloadTrackerSQLite.get_tracker_path("cbz", "./Tamamo no Koi")
db = sqlite3.connect(db_path)

cls: SQLMigration
for cls, file in _iter_migrate_cls():
    if file != "00000_init.py":
        continue

    print(cls.check_if_migrate_is_possible())
    cls.migrate()

tree

.
├── Tamamo no Koi
│   ├── cover.jpg
│   └── download.db
└── script.py

2 directories, 3 files

python script.py

Traceback (most recent call last):
  File "/downloads/script.py", line 6, in <module>
    db_path = DownloadTrackerSQLite.get_tracker_path("cbz", "./Tamamo no Koi")
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py", line 147, in get_tracker_path
    return path / "download.db"
           ~~~~~^~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for /: 'str' and 'str'
vonProteus commented 3 months ago

after changing /usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py 147 to return path + "/download.db" and changing script to 

import sqlite3
from mangadex_downloader.tracker.sql_migrations import _iter_migrate_cls
from mangadex_downloader.tracker.sql_migrations.base import SQLMigration
from mangadex_downloader.tracker.sqlite import DownloadTrackerSQLite

db_path = DownloadTrackerSQLite.get_tracker_path("cbz", "./Tamamo no Koi")
db = sqlite3.connect(db_path)

cls: SQLMigration
for cls, file in _iter_migrate_cls(db):
    if file != "00000_init.py":
        continue

    print(cls.check_if_migrate_is_possible())
    cls.migrate()

first run is True second is False

mansuf commented 3 months ago

after changing /usr/local/lib/python3.11/site-packages/mangadex_downloader/tracker/sqlite.py 147 to return path + "/download.db" and changing script to

Sorry i was forgot to fix that, it was actually 

from pathlib import Path
db_path = DownloadTrackerSQLite.get_tracker_path("cbz", Path("./Tamamo no Koi"))

first run is True second is False

Huh, if the first run is True and and the second one is False then the migration is success. Is the database file (download.db) size is increased after that ?

vonProteus commented 3 months ago 
root@77078be60e1c:/downloads# ls -alh Tamamo\ no\ Koi/
total 272K
drwxr-xr-x 2 root root 4.0K Jun  6 15:43 .
drwxr-xr-x 1 root root 4.0K Jun  6 15:43 ..
-rw-r--r-- 1 root root 255K Jun  6 15:43 cover.jpg
-rw-r--r-- 1 root root 8.0K Jun  6 15:43 download.db
root@77078be60e1c:/downloads# python script.py
True
root@77078be60e1c:/downloads# ls -alh Tamamo\ no\ Koi/
total 284K
drwxr-xr-x 2 root root 4.0K Jun  6 15:44 .
drwxr-xr-x 1 root root 4.0K Jun  6 15:44 ..
-rw-r--r-- 1 root root 255K Jun  6 15:43 cover.jpg
-rw-r--r-- 1 root root  20K Jun  6 15:44 download.db
mansuf commented 3 months ago

Okay, so the download.db size is increased and the migration is success. And why the heck it's not working if the app is running normally ?

I don't know man, i'm really confused right now. Either the application has a bug that only you can reproduce it or something going on with the docker image or permissions files. You have to fix it yourself, i'm really sorry i cannot provide solution any further.

If you found a fix for the bug you can tell me and i will create documentaion of how to fix this issue or patch to v3.0.0 branch.

For now you might want to apply 00000_init.py migration manually from script that i give to you https://github.com/mansuf/mangadex-downloader/issues/110#issuecomment-2148918844.

vonProteus commented 3 months ago

i don't know why but when i do this

diff --git a/mangadex_downloader/tracker/sql_migrations/00000_init.py b/mangadex_downloader/tracker/sql_migrations/00001_init.py
rename from mangadex_downloader/tracker/sql_migrations/00000_init.py
rename to mangadex_downloader/tracker/sql_migrations/00001_init.py
diff --git a/mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py b/mangadex_downloader/tracker/sql_migrations/00002_add_table_db_info_and_alter_table_file_info.py
rename from mangadex_downloader/tracker/sql_migrations/00001_add_table_db_info_and_alter_table_file_info.py
rename to mangadex_downloader/tracker/sql_migrations/00002_add_table_db_info_and_alter_table_file_info.py

this seems to solve the problem

mansuf commented 2 months ago

i don't know why but when i do this

Hmmmm, interesting. It's really works just like that ? I have created that patch https://github.com/mansuf/mangadex-downloader/commit/ffd2f24463fb8904fb6da61973b108d5ef1b1be0, if you can test it, it could be useful

vonProteus commented 2 months ago

first run from https://github.com/mansuf/mangadex-downloader/commit/ffd2f24463fb8904fb6da61973b108d5ef1b1be0

time (nocorrect docker build --progress=plain -t mangadex-downloader . && docker run -ti -v ./downloads:/downloads/ --rm mangadex-downloader "tamamo no koi" -s --input-pos 1 -f cbz --log-level=DEBUG )

DEBUG] Config 'stacked_progress_bar_order' (value: ['volumes', 'chapters', 'pages', 'file sizes', 'convert']) is not passed validator test, reason: 'list' object has no attribute 'split'. Falling back to default value
[DEBUG] Loaded config from cli args = {'login_cache': False, 'language': 'en', 'cover': 'original', 'save_as': 'cbz', 'use_chapter_title': False, 'use_compressed_image': False, 'force_https': False, 'path': './{manga.title}', 'filename_chapter': '{chapter.simple_name}{file_ext}', 'filename_volume': 'Vol. {volume}{file_ext}', 'filename_single': 'All chapters{file_ext}', 'dns_over_https': None, 'no_group_name': False, 'sort_by': 'volume', 'no_progress_bar': False, 'http_retries': 5, 'write_tachiyomi_info': False, 'download_mode': 'default', 'use_chapter_cover': False, 'use_volume_cover': False, 'no_track': False, 'volume_cover_language': None, 'stacked_progress_bar_order': 'volumes, chapters, pages, file sizes, convert', 'log_level': 'DEBUG', 'progress_bar_layout': 'default', 'ignore_missing_chapters': False, 'create_no_volume': False, 'create_manga_info': False, 'manga_info_format': 'csv', 'manga_info_filepath': '{download_path}/manga_info.{manga_info_format}'}
[INFO] Checking url = 433e77d2-5a58-48a3-95b8-e3c02f309255
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/Tamamo no Koi"
[INFO] Using cbz format
[INFO] Downloading cover manga Tamamo no Koi
File sizes: 100%|████████████████████████████| 260k/260k [00:00<00:00, 1.82MB/s]
[DEBUG] Applying download tracker database migration 00001_init.py...
[DEBUG] Applying download tracker database migration 00002_add_table_db_info_and_alter_table_file_info.py...
[INFO] Preparing to download...
[INFO] Getting images from chapter 0
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 1
File sizes: 100%|█████████████████████████████| 695k/695k [00:00<00:00, 843kB/s]
[DEBUG] Reporting {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/2cb2a15b58b96bc72b705849ce15ed8b/1-42b91b72bbce2b2f62c24c4c4d56a84d79bc6952139e19924a7a6acc72ed13eb.png', 'success': True, 'cached': False, 'bytes': 694683, 'duration': 1277} to MangaDex network
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 2
File sizes: 100%|████████████████████████████| 536k/536k [00:00<00:00, 3.38MB/s]
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 3
File sizes: 100%|████████████████████████████| 491k/491k [00:00<00:00, 3.69MB/s]
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 4
File sizes: 100%|████████████████████████████| 457k/457k [00:00<00:00, 3.61MB/s]
[INFO] Downloading [AteTeenPlus & AfterLife Scans] Chapter. 0 Oneshot page 5
[DEBUG] Successfully send report {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/2cb2a15b58b96bc72b705849ce15ed8b/1-42b91b72bbce2b2f62c24c4c4d56a84d79bc6952139e19924a7a6acc72ed13eb.png', 'success': True, 'cached': False, 'bytes': 694683, 'duration': 1277} to MangaDex network
[DEBUG] Reporting {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/2cb2a15b58b96bc72b705849ce15ed8b/2-b6531c9af440614fcb55854eeef444f53956b7df7ca87d6e691f8d347a399826.png', 'success': True, 'cached': False, 'bytes': 535864, 'duration': 204} to MangaDex network
…
[INFO] Downloading [No Group Scanlation] Volume. 1 Chapter. 6 page 31
[DEBUG] Successfully send report {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/a146b42e90bda4200e616215027ba343/30-9a0448957fadd95d73c962e5cac07a5da12ea6f68990631845f74e5243b7e3c2.jpg', 'success': True, 'cached': False, 'bytes': 355792, 'duration': 1634} to MangaDex network
File sizes: 100%|█████████████████████████████| 351k/351k [00:00<00:00, 412kB/s]
[DEBUG] Reporting {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/a146b42e90bda4200e616215027ba343/31-56bf4d39c94e1dd222849bbf6c98065a7afb758214a12bbc8679ffe12a80fccf.jpg', 'success': True, 'cached': False, 'bytes': 350860, 'duration': 2005} to MangaDex network
[INFO] Downloading [No Group Scanlation] Volume. 1 Chapter. 6 page 32
[DEBUG] Successfully send report {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/a146b42e90bda4200e616215027ba343/31-56bf4d39c94e1dd222849bbf6c98065a7afb758214a12bbc8679ffe12a80fccf.jpg', 'success': True, 'cached': False, 'bytes': 350860, 'duration': 2005} to MangaDex network
File sizes: 100%|█████████████████████████████| 221k/221k [00:00<00:00, 933kB/s]
[DEBUG] Reporting {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/a146b42e90bda4200e616215027ba343/32-d337c78d7b14f9c4f3f38968cef2b57e0b719865d59be8657beb0f18549f4195.png', 'success': True, 'cached': False, 'bytes': 221338, 'duration': 1443} to MangaDex network
[INFO] [No Group Scanlation] Vol. 1 Ch. 6 has finished download, converting to .cbz file...
Convert: 100%|██████████████████████████████| 32.0/32.0 [00:00<00:00, 128page/s]
[INFO] Waiting for chapter read marker to finish
[INFO] Download finished for manga "Tamamo no Koi"
[DEBUG] Checking update...
[DEBUG] Successfully send report {'url': 'https://cmdxd98sb0x3yprd.mangadex.network/data/a146b42e90bda4200e616215027ba343/32-d337c78d7b14f9c4f3f38968cef2b57e0b719865d59be8657beb0f18549f4195.png', 'success': True, 'cached': False, 'bytes': 221338, 'duration': 1443} to MangaDex network
[DEBUG] No update found
[INFO] Cleaning up...
[DEBUG] Closing network object

and second try

[DEBUG] Config 'stacked_progress_bar_order' (value: ['volumes', 'chapters', 'pages', 'file sizes', 'convert']) is not passed validator test, reason: 'list' object has no attribute 'split'. Falling back to default value
[DEBUG] Loaded config from cli args = {'login_cache': False, 'language': 'en', 'cover': 'original', 'save_as': 'cbz', 'use_chapter_title': False, 'use_compressed_image': False, 'force_https': False, 'path': './{manga.title}', 'filename_chapter': '{chapter.simple_name}{file_ext}', 'filename_volume': 'Vol. {volume}{file_ext}', 'filename_single': 'All chapters{file_ext}', 'dns_over_https': None, 'no_group_name': False, 'sort_by': 'volume', 'no_progress_bar': False, 'http_retries': 5, 'write_tachiyomi_info': False, 'download_mode': 'default', 'use_chapter_cover': False, 'use_volume_cover': False, 'no_track': False, 'volume_cover_language': None, 'stacked_progress_bar_order': 'volumes, chapters, pages, file sizes, convert', 'log_level': 'DEBUG', 'progress_bar_layout': 'default', 'ignore_missing_chapters': False, 'create_no_volume': False, 'create_manga_info': False, 'manga_info_format': 'csv', 'manga_info_filepath': '{download_path}/manga_info.{manga_info_format}'}
[INFO] Checking url = 433e77d2-5a58-48a3-95b8-e3c02f309255
[INFO] Using English language
[INFO] Fetching all chapters...
[INFO] Download directory is set to "/downloads/Tamamo no Koi"
[INFO] Using cbz format
[INFO] Downloading cover manga Tamamo no Koi
[INFO] File exist and replace is False, cancelling download...
[DEBUG] Applying download tracker database migration 00002_add_table_db_info_and_alter_table_file_info.py...
[INFO] Preparing to download...
[INFO] Verifying downloaded chapters...
[INFO] 'Ch. 0 Oneshot.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 1.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 2.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 3.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 4.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 5.cbz' is verified and no need to re-download
[INFO] 'Vol. 1 Ch. 6.cbz' is verified and no need to re-download
[INFO] Waiting for chapter read marker to finish
[INFO] Download finished for manga "Tamamo no Koi"
[DEBUG] Checking update...
[DEBUG] No update found
[INFO] Cleaning up...
[DEBUG] Closing network object

so everything looks fixed now

mansuf commented 2 months ago

A simple rename files is key to fixing bugs, yeah.

Anyway, thank you for your contribution from testing, reporting bugs and everything !