Testbed - Githubissues

Introduction of a Possible PanelCleaner Testbed

Overview

This Pull Request establishes the _testbed directory within the PanelCleaner project. It is designed as a separate space for developers and maintainers to experiment with and test new ideas, technologies, and methodologies that could influence future directions of PanelCleaner.

Key Features

Dedicated Exploratory Environment: The _testbed serves as a sandbox for trying out new DL technologies and frameworks, currently focusing on Tesseract and IDefics models OCR perfomance.
Experiment Framework: I have begun to build an experiment framework that will assist in assessing the effectiveness of new DL technologies and models.
Google Colab Integration: To facilitate easy access and collaboration, the testbed is intended to be fully compatible with Google Colab, allowing experiments to be run directly on the platform.
Comprehensive Documentation: An initial barebones README.md has been added to guide developers through the setup, usage, and contribution processes. It includes installation instructions, an introduction to the nbdev literate programming environment, and a summary of each included notebook.
Structured Experimentation: Notebooks such as ocr_metric.ipynb and experiments.ipynb are included to define metrics and document the evaluation processes.

Impact

The creation of the _testbed directory can be a move to enhance the development process of PanelCleaner and allow collaborations by providing a structured environment for innovation and testing. This will aid in quickly iterating over new ideas and integrating successful ones into the main project.

Future Directions

The testbed should evolve based on the experiments conducted and the feedback from developers. It will continuously adapt to include more sophisticated tools and frameworks that align with the evolving field of deep learning.

Initial Commit of the Testbed: There are likely several bugs and issues at this stage, but I wanted to avoid further delays. I plan to test the Google Colab integration tomorrow and will continue refining the setup based on the results.

Since the previous PR is now merged, I did a quick rebase here and it looks like that all worked no problem. I'll be interested in seeing where this goes!

Well, about that 80/20 rule—80% of the code in 20% of the time? In my case, it's more like 10/90. I had most of the current functionality nailed down in the previous commit, everything working as intended on Linux. However, the assumption that Colab wouldn't be a problem turned out to be very wrong.

Colab requires Python 10, while PanelCleaner is using Python 11. Image visualization in Colab is unacceptably slow, and this commit uses a lot of them. In general, everything was very slow and/or very buggy.

I've had to refactor most of the code before time, move the _testbed folder inside pcleaner, and even implement a web server with a reverse proxy just to make the Colab experience smooth enough.

Idefics quantized to 4 bits can run in most consumer-grades cards at or above 6GB and Colab free tier.

I think the result is more robust, useful, and efficient. The framework is now in a state where it's much easier to add new experiments and tests.

This was work, and now the (hopefully) fun part:

Future Work (No particular order or schedule):

Bug Hunting: Surely there's a lot to find.
OCR Metrics Improvement: Particularly, adding Levenshtein distance.
Experiment Comparisons: Both for runs in one experiment and across different experiments.
Report Generation: In HTML and PDF.
Ground Truth Enhancements:
- Improve existing ones. I've added a few but only to advance a bit. Many images in the standard set don't have one or are just results from one experiment. I plan to add simple editing capabilities.
- I'm unable to check or even generate ground truth for Japanese. If someone could help generate them, that would be great. Maybe using manga-ocr as ground truth would be enough. Also, consider adding images in other languages.
- Decide if we want to check OCR with all-caps or not. In manga, the point is moot, but for comics, the OCR performance using all-caps or properly capitalized text is considerable.
New OCR Models: These last weeks have seen the publishing of two very interesting models: PaliGemma from Google and Phi-3 Vision from Microsoft. Both are slimmer than Idefics and very promising for OCR in preliminary testing.
LLM Support: Add support for GPT-4o (now free), GPT-4, Claude and/or Gemini (paid), and Llama-3 (free). Not only for their text efficiency, which makes them useful for OCR and/or translation, but also for their vision capabilities. Many PanelCleaner users can't or don't want to use these models, but for pro users, I'm sure they'll have no qualms about using them. In any case, we all end up using them one way or another.
Fine-Tuning/Synthetic Datasets: Idefics and the other vision models are designed to be fine-tuned for downstream tasks. Thus we need appropriate datasets of which we don't have any. Since Manga 109 is not accessible and the comics datasets are few and small, and the people from comic-text-detector probably won't have time to help us, we have to generate a synthetic dataset of manga/comics. It's not an ideal solution, but it will serve to fine-tune and test the limits of the models. And it's easy to generate.
New PanelCleaner Functionality Testing:
- Ordering: Idefics, PaliGemma, and Phi-3 are not OCR models per se, they only have OCR functionality among many others that put them in the field of Visual Document Understanding (VDU), that is, you can chat with images and get text. I want to test if they can detect panels and their reading order. Claude, in particular, is very good with structured output and can generate JSON readily.
- Text and Panel Detection and Segmenting: Of high, production quality. Probably for that, we'll need more specialized models like SAM (Segment Anything) or YOLO.

Wow, got a lot of plans, impressive!

All those technical issues getting collab to work sound like an absolute nightmare. You have my condolences.

About generating reports, I've got some experience generating LaTeX documents to make pdfs, then transforming that into html. But if the aesthetics aren't super important, then I'd definitely recommend generating HTML, then using a python package to dump the html into a pdf file automatically. It may not look amazing, but takes no effort.

Let me know if you run into issues that need my attention, or would like to share some thought.

Thank you for your insights on the report generation! I'll definitely consider the HTML to PDF route for simplicity.

Could you please review the two final user (for developers, really) notebooks, test_tesseract.ipynb and test_idefics.ipynb, on your Linux system or in Colab? I’d appreciate any feedback on these as they are key to the testbed. The others in the nbs folder are development ones and might be too involved for a quick look.

If you’re interested and have some time, the experiments.ipynb notebook is also quite insightful. It’s structured to be followed sequentially and illustrates the main concepts I’m working with.

I've successfully integrated the Paligemma model—amazing little model, remarkable performance and accuracy, pity about the restrictive license. It's outperforming Idefics2 and has achieved 98-99% accuracy without fine-tuning. This will likely be the last OCR model I test for a while as I shift focus to other functionalities.

I’ll dive into the reporting aspect soon, but first, there’s some boring bookkeeping, like run comparisons, I need to tackle.

Looking forward to your thoughts and any improvements you might have!

Wow, the ExperimentsVisor is super cool! It took a bit to get vc code to like the venv and then I struggled a bit to get it to load the latest test version. I couldn't get it to load from source, but running make fresh-install with the venv I wanted active got a version with everything loaded. It'd be better to figure out how to load it from source directly, not like that's applicable to colab anyway though. Isn't getting that to work the whole reason you moved _testbed into pcleaner though? Maybe I'm just too incompetent at jupyter notebooks (I typically avoid them).

The os.chdir(package_path/'_testbed') would put the path inside the site-packages folder, which isn't so great, so I manually bent that path to point to pcleaner/_testbed/ and it worked.

I tried the idefics too, but unfortunately got stuck here:

{
    "name": "TypeError",
    "message": "'<' not supported between instances of 'NoneType' and 'int'",
    "stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[21], line 1
----> 1 idefics_experiment = ExperimentsVisor(
      2                         CONTEXT,
      3                         'Idefics',
      4                         image_idx='Strange_Tales_172005.jpg',
      5                         box_idx=1,
      6                         method=CropMethod.DEFAULT_GREY_PAD
      7                     )
      8 idefics_experiment

File ~/Repos/PanelCleaner/venv/lib/python3.11/site-packages/pcleaner/_testbed/testbed/experiments.py:2263, in ExperimentsVisor.__init__(self, ctx, ocr_model, root_dir, config, image_idx, run_name, run_names, display_option, all_boxes, box_idx, all_methods, method, out)
   2261 if ctx is None:
   2262     ctx = self.setup_experiment_context(ocr_model, root_dir, config, run_names)
-> 2263 _ = ImageContext(ctx, image_idx)  # raises if image_idx is out of range
   2265 out = out or self.out
   2266 run_selector = RunSelector(ctx, run_name, run_names, out=out)

File ~/Repos/PanelCleaner/venv/lib/python3.11/site-packages/pcleaner/_testbed/testbed/experiments.py:639, in ImageContext.__new__(cls, exp, idx, *args, **kwargs)
    635 def __new__(cls,
    636         exp: ExperimentContext,
    637         idx: ImgSpecT,
    638         *args, **kwargs) -> 'ImageContext':
--> 639     return super().__new__(cls, exp, idx, *args, **kwargs)

File ~/Repos/PanelCleaner/venv/lib/python3.11/site-packages/pcleaner/_testbed/testbed/experiments.py:300, in ExperimentSubject.__new__(cls, exp, idx, *args, **kwargs)
    298 if self is None:
    299     self = super().__new__(cls)
--> 300     self = exp.setup_subject_context(idx, self, *args, **kwargs)
    301     if self is None:
    302         raise ValueError(f\"Can't create new subject with idx: {idx}: out of range\")

File ~/Repos/PanelCleaner/venv/lib/python3.11/site-packages/pcleaner/_testbed/testbed/experiments.py:463, in ExperimentContext.setup_subject_context(self, idx, subject, *args, **kwargs)
    461 def setup_subject_context(self, idx: SubjIdT, /, subject: ExperimentSubject, *args, **kwargs):
    462     \"Setup and set cached subject.\"
--> 463     if idx < 0 or idx >= self.subject_count:
    464         raise ValueError(f\"Can't create new subject with idx: {idx}: out of range\")
    465     subject.setup(self, idx, *args, **kwargs)

TypeError: '<' not supported between instances of 'NoneType' and 'int'"
}

And with that being in an auto generated 2k lines file (wow) I didn't figure out a way to fix it.

Crazy impressive though! Can't wait to see Idefics in action for myself.

Side note: Pinning a specific Pillow version will not be acceptable for the final build, as it'll cause unnecessary conflicts for anyone installing it at a system level, just gotta remember to remove that when not testing for colab.

One release later, I got the latest patches on master over here. Hope it doesn't cause conflicts.

All platforms except Flatpak have your awesome update...I don't have time to open that can of worms today. Thanks again for all the work :)

Oops, just like with the first commit, I tested the Colab setup but overlooked the local install, which seems to have some issues.

I'm currently in the middle of a refactor, but I'll push a commit with improved documentation and some fixes later today or tomorrow.

The os.chdir(package_path/'_testbed') would put the path inside the site-packages folder, which isn't ideal, so I manually adjusted that path to point to pcleaner/_testbed/ and it worked.

Absolutely, that line was intended for a Colab local dev install. I forgot to wrap it with if IN_COLAB: .... Thanks for catching that!

I tried the idefics too, but unfortunately got stuck here:

Not sure what the issue is yet, but I'm planning to strengthen and modify both test_idefics.ipynb and ocr_idefics.ipynb/py in the next commit. Hopefully, that will resolve the issue.

Pinning a specific Pillow version will not be acceptable for the final build, as it'll cause unnecessary conflicts for anyone installing it at a system level, just gotta remember to remove that when not testing for Colab.

I completely agree about not pinning requirements. The issue with PIL in Colab is peculiar, though. It seems related to imageio, which, as far as I know, PanelCleaner doesn't use. I haven't had the chance to look into this problem yet.

If we decide to continue with this PR and the testbed playground, we might consider creating a CLI-only package/wheel for installing PanelCleaner in Colab with leaner requirements. In Colab, we really only need the CLI components.

By the way, it appears entirely feasible to launch PySide6/QT from Jupyter and simultaneously have the GUI, the CLI, and everything running in a notebook environment sharing the kernel and all the debugging tools. I haven't tested this yet, though. Right now, I'm more into the CLI than the GUI part of PanelCleaner.

Sorry for the inconvenience. As a bonus, in the next commit, I'll include improvements to both Tesseract and Idefics, along with the addition of the awesome PaliGemma.

I tried the idefics too, but unfortunately got stuck here:

Fixed the issue. The visor wasn't finding the test images. It's simply a matter of catching the fact earlier and informing the user

That sounds awesome, looking forward to it.

I'm not sure there is much to slim down in terms of having a CLI only build, I think it's just PySide6. It'd save 530MB in the unpacked state from the total of 2.1GB without CUDA, or the total of 7.3GB with CUDA, not to mention the 31GB of idefics, but at least pip doesn't need to deal with those.

All of the assets and code for the GUI are a mere 2.7MB.

I played around with the idea and it's actually trivial to generate such a whl, much to my surprise. The whl is significantly smaller, 400kB vs 3.5MB. Without dependcies. So let me know if saving 500MB makes a difference and if I forgot some dependencies that are gui-exclusive.

I've updated the README with detailed instructions for local installation from scratch and thoroughly tested it on both mac and Linux to ensure everything is in good shape.

I've also streamlined the Colab integration, which now doesn't require a full PanelCleaner. This did require a few tweaks outside the testbed folder however, specifically to the tr() function in helpers and the manga-ocr import. It's just a temporary fix to smooth things over for Colab users.

This got me thinking about the possibility of a CLI-only install of PanelCleaner for Colab, which could be a neat solution for users without GPUs.

I've also added PaliGemma, which seems to be a friendlier and more powerful model than Idefics, especially for OCR tasks.

There have been a lot of changes and refactors aimed at stabilizing the current build, even though it's still somewhat limited. These updates should really help smooth the way for next steps.

Please try installing clean and experimenting with the testbed both locally and in Colab to further refine the process and catch any bugs that may already be lurking in the shadows.

Oops, forgot to mention the ground truth file format has changed. Re-download the source folder or delete the .txt files and add the new JSONs.

I'm not sure there is much to slim down in terms of having a CLI only build, I think it's just PySide6. It'd save 530MB in the unpacked state from the total of 2.1GB without CUDA, or the total of 7.3GB with CUDA, not to mention the 31GB of idefics, but at least pip doesn't need to deal with those.

All of the assets and code for the GUI are a mere 2.7MB.

I played around with the idea and it's actually trivial to generate such a whl, much to my surprise. The whl is significantly smaller, 400kB vs 3.5MB. Without dependcies. So let me know if saving 500MB makes a difference and if I forgot some dependencies that are gui-exclusive.

So, I suppose this would help then? Because I already got it working. It'd generate a separate whl file, which are what pip installs, completely without gui stuff.

Looking like it was a busy and successful week!

Hey Voxel,

Glad that you're even considering the Colab stuff, overcoming your little love of notebooks 😇.

Yeah, an option allowing us to do: pip install pcleaner-CLI would be all we need to get PanelCleaner up and running in one minute instead of several.

Your findings about the significant reduction in the wheel file size are really promising! I'm no expert on python packages or wheels.

Regarding the inclusion of dependencies, you're right that we don't need to bundle all the dependencies directly with the wheel. For a CLI-only tool, I think that is standard practice, isn't it? For apps and exes probably is a good idea to bundle all the dependencies. For a command-line tool, like pcleaner-CLI would be, pip will automatically handle the installation of these necessary dependencies without us needing to package them directly. Maybe we don't even need wheel? I don't know, just asking.

I'm not sure if you've had a chance to try out Colab yet, but I'd really appreciate your thoughts on its usability and overall experience. The current setup is a bit hackish, I know, but here are the timings:

Mounting GDrive (optional): 15s
PanelCleaner setup: 8-10s
HuggingFace accelerate (needed for PaliGemma): 1 min; Idefics needs some more HF libraries for quantization.
Testbed stuff (potentially mainstream in the future): 15s

Model load times vary. comic-text-detector loads quickly, while Tesseract takes a couple of minutes due to multiple steps. PaliGemma is very fast in all versions. Idefics is currently slower because we have to download the fine-tuned version each time. If we decide to integrate it into the main PanelCleaner build, we could pre-generate the 3 quantization, upload them to HuggingFace, and significantly reduce load times. I also tested downloading the models to the user's Google Drive but found it slower than local downloads.

In about 1.5 minutes, you can start working with PanelCleaner/testbed using a more or less powerful GPU from anywhere you have a browser on a PC, phone, or tablet. Besides Colab, there are many other free and paid notebook-based services like Kaggle, Azure, IBM Watson, Google Cloud, Amazon SageMaker, Databricks, etc.

IMHO, I think it would be beneficial to have Colab support. That's the whole point of the testbed subpackage—to test new ideas not only for performance but also for their implications for inclusion in PanelCleaner main. However, you have to consider the audience of PanelCleaner. From my perspective, PanelCleaner targets two types of users:

Manga and/or comics fans looking for tools to help with manga/comic book adaptations, and professionals like FujiCrow searching for help with commercial book production.
Developers like me, who are excited about the brave new world of AI. There's probably some overlap between these two groups. Developers are likely the first to discover PanelCleaner while looking for implementations of DL models.

Both groups would greatly benefit from notebook support. It would also help with the discovery of PanelCleaner. Imagine a Colab badge in the README or on the project page that launches PanelCleaner, ready to clean manga and showcasing all its capabilities before trying the app.

And, on a personal note, as my Linux setup lacks a window manager, a CLI-only package would be much appreciated.

How to proceed? I would open a new PR and explore four levels of integration.

Before we get into that, let's review the options. This explanation will be long, so please bear with me. Note that in this discussion, I am not considering model downloads, which are unavoidable for non-local installations and must be performed each time.

The libraries pre-installed in Colab are specifically tailored for data scientists and DL research, which might differ from other notebook platforms. I've already streamlined the PanelCleaner CLI requirements to exclude those already present in Colab (see requirements-colab.txt). This is why it installs so quickly now. This efficiency is also due to minimizing the use of the notoriously slow pip. While pip is not the most efficient package manager, it performs many checks that are necessary for ensuring compatibility and stability.

Current solution with git, 12-15 seconds: (Ugly and hackish, but fast)

if FC.IN_COLAB:
    pc_path = 'PanelCleaner'
    tb_path = pc_path/'pcleaner/_testbed'
    !git clone -b testbed https://github.com/civvic/PanelCleaner.git
    assert tb_path.exists(), "PanelCleaner not found"
    os.chdir(tb_path)
    sys.path.append(f"{pc_path.resolve()}")
    sys.path.append(f"{tb_path.resolve()}")
    !pip install -q -r requirements-colab.txt

Using pip, > 2 minutes: (Colab is Docker with Ubuntu 18.04, so this will be similar in any other linux)

!pip install git+https://github.com/civvic/PanelCleaner.git@testbed

Collecting git+https://github.com/civvic/PanelCleaner.git@testbed
  Cloning https://github.com/civvic/PanelCleaner.git (to revision testbed) to /tmp/pip-req-build-7qvv_1fq
  Running command git clone --filter=blob:none --quiet https://github.com/civvic/PanelCleaner.git /tmp/pip-req-build-7qvv_1fq
  Running command git checkout -b testbed --track origin/testbed
  Switched to a new branch 'testbed'
  Branch 'testbed' set up to track remote branch 'testbed' from 'origin'.
  Resolved https://github.com/civvic/PanelCleaner.git to commit 59177340547b0f3699d4396b19e8c9e5b3b012ad
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (4.8.0.76)
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (4.41.1)
Collecting manga-ocr (from pcleaner==2.7.0)
  Downloading manga_ocr-0.1.11-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.9/62.9 kB 2.0 MB/s eta 0:00:00
Requirement already satisfied: Pillow==9.4.0 in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (9.4.0)
Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (2.3.0+cu121)
...
Requirement already satisfied: tifffile in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (2024.5.22)
Collecting PySide6>=6.7.0 (from pcleaner==2.7.0)
  Downloading PySide6-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (530 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 530.7/530.7 kB 30.3 MB/s eta 0:00:00
Requirement already satisfied: attrs in /usr/local/lib/python3.10/dist-packages (from pcleaner==2.7.0) (23.2.0)
...
Collecting simple-lama-inpainting (from pcleaner==2.7.0)
  Downloading simple_lama_inpainting-0.1.2-py3-none-any.whl (9.6 kB)
...
Collecting shiboken6==6.7.1 (from PySide6>=6.7.0->pcleaner==2.7.0)
  Downloading shiboken6-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (188 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.4/188.4 kB 18.4 MB/s eta 0:00:00
Collecting PySide6-Essentials==6.7.1 (from PySide6>=6.7.0->pcleaner==2.7.0)
  Downloading PySide6_Essentials-6.7.1-cp39-abi3-manylinux_2_28_x86_64.whl (87.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.6/87.6 MB 8.3 MB/s eta 
0:00:00
...
  Preparing metadata (setup.py) ... done
Collecting fugashi (from manga-ocr->pcleaner==2.7.0)
  Downloading fugashi-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (600 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 600.9/600.9 kB 34.3 MB/s eta 0:00:00
Collecting jaconv (from manga-ocr->pcleaner==2.7.0)
  Downloading jaconv-0.3.4.tar.gz (16 kB)
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of manga-ocr to determine which version is compatible with other requirements. This could take a while.
Collecting manga-ocr (from pcleaner==2.7.0)
  Downloading manga_ocr-0.1.10-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 6.1 MB/s eta 0:00:00
Requirement already satisfied: pyperclip in /usr/local/lib/python3.10/dist-packages (from manga-ocr->pcleaner==2.7.0) (1.8.2)
Collecting unidic-lite (from manga-ocr->pcleaner==2.7.0)
  Downloading unidic-lite-1.0.8.tar.gz (47.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.4/47.4 MB 11.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch->pcleaner==2.7.0) (3.14.0)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch->pcleaner==2.7.0) (4.11.0)
...
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch->pcleaner==2.7.0)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch->pcleaner==2.7.0)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
...
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 51.8 MB/s eta 0:00:00
Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers->pcleaner==2.7.0) (0.23.1)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers->pcleaner==2.7.0) (24.0)
...
Collecting fire (from manga-ocr->pcleaner==2.7.0)
  Downloading fire-0.5.0.tar.gz (88 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.3/88.3 kB 10.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of simple-lama-inpainting to determine which version is compatible with other requirements. This could take a while.
Collecting simple-lama-inpainting (from pcleaner==2.7.0)
  Downloading simple_lama_inpainting-0.1.1-py3-none-any.whl (9.6 kB)
  Downloading simple_lama_inpainting-0.1.0-py3-none-any.whl (13 kB)
Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from fire->manga-ocr->pcleaner==2.7.0) (1.16.0)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire->manga-ocr->pcleaner==2.7.0) (2.4.0)
...
Building wheels for collected packages: pcleaner, fire, jaconv, unidic-lite
  Building wheel for pcleaner (pyproject.toml) ... done
  Created wheel for pcleaner: filename=pcleaner-2.7.0-py3-none-any.whl size=1349831 sha256=272b31f9f54c164950f83514612d8b26fdf5870a45530743940183e4997393c8
  Stored in directory: /tmp/pip-ephem-wheel-cache-f4rz_d29/wheels/84/e3/8c/b198fffc383f392c64a5c379c7c252a362221e00be0d0160e2
  ...
Successfully built pcleaner fire jaconv unidic-lite
Installing collected packages: unidic-lite, pyclipper, jaconv, dictdiffer, xdg, shiboken6, python-magic, pytesseract, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, ...
...

Of those 2 minutes, more than half is for manga-ocr. Installing manga-ocr by itself is 1 min+. pip also wastes a lot of time checking requirements already installed. As I'm not installing manga-ocr I cheated. A fairer comparison would be 1 min+ with pip vs 12-15s with the hack. And, of course, the hack is more brittle, Google can change Colab pre-installed libraries any time.

Probably, but have to check, it would be faster to install the two models, manga-ocr and simple-lama-inpainting, by themselves apart from the rest of the CLI, i.e.,

pip install manga-ocr
pip install simple-lama-inpainting
pip install pcleaner-CLI

So, have an option with only the reqs for Colab, pcleaner-CLI-colab, or a more general that can be installed anywhere, pcleaner-CLI or both or neither?

Apart from that decision to take beforehand, the next phases in a possible new PR would be:

Basic support: Adapt the CLI for Colab/Notebook use. This should be straightforward—just a matter of adapting the current CLI help to a notebook format with examples and writing a couple of notebooks to play with the API. Maybe add some visualization helpers, nothing too fancy. Two points of contention here:
- Localization Support: Decide what to do with the tr() function. For the initial version, we might only support English to keep things simple. I'm not sure if there are straightforward alternatives to PySide6/Qt GUI translation services.
- Dependency Management: Investigate the manga-ocr requirement to see why they have pinned the 10.x version of PIL. It's hard to believe they're using some functionality that wasn't in the 9.x release. The Colab team is slow, but they'll probably upgrade PIL (and Python 10) soon.
Beyond Colab: I don't know how feasible is for us testing all those platforms to see which are its specific requirements. To much work and the type that isn't fun.
Optional but nice to have: Leverage some testbed functionality and use/adapt/create some widgets like the ones I've made for the testbed notebooks, so users learning about PanelCleaner don't have to write code to work with it. The widgets are designed to be composed effortlessly.
Also optional and more complex: Write a Gradio widget. All DL practitioners know that HuggingFace Spaces are a great way to share and showcase your work. It's very familiar and widely used. Gradio can be used in Colab and any notebook and deployed locally or remotely besides HF spaces. I'm not a fan of Gradio—it's a good idea I don't like how is implemented—but it's widely used now, with thousands of new spaces continuously appearing on HuggingFace showcasing new papers and models.

In my view, PanelCleaner is a specialized tool for manga/comics processing that leverages deep learning. You surely have a different perspective, and as the creator, your vision is what matters. Nonetheless, PanelCleaner has the potential to evolve into a multi-platform tool that caters to diverse needs in the specialized domain of comics, offering various interfaces like command line, GUI, and notebooks. This evolution should be gradual and pressure-free, this is a hobby.

Before all that, it's important to acknowledge the current state of PanelCleaner. It's a robust, self-contained package that's stable and performs well, designed with specific functionalities and audiences in mind. Expanding its scope is certainly exciting and fun work, but it also introduces potential for more bugs, more issues, and increased support demands. We need to consider whether we are willing or able to dedicate more time to new stuff. Maintaining open-source packages is rewarding yet demanding.

Too ambitious? Am I getting ahead of myself?

BTW, I plan to include manga-ocr in the testbed soon for comparison purposes. I'll check for any dependency collisions at that time.

Also, could you spare a few minutes to help create the ground truth for one or two of the Japanese pages in the testbed set? I'm not quite sure how to start with that.

Also, I'm no expert in manga as your are, maybe we should include more representative pages than the ones I selected in a hurry.

Whoa, that's a lotta info.

Yes, then I'll be pushing my changes to have a make build-cli that produces a pcleaner-cli. (There will be conflicts, I'll clean them up)

It is important to understand what a whl is. This is a zip file containing the source code and some metadata that pip understands so it can install the package for you. Wheels do not contain dependencies (unless you actually just pasted in the source code, as I had to for comic-text-detector), they just tell pip what you need. So the wheel being smaller is just that all the gui code and icon data isn't included.

The bigger win is no longer needing PySide6, which saves 530MB. The remaining dependencies stil need 1.5GB, or roughly 7GB if CUDA. So the savings aren't really that big.

If you wish to limit dependency overlap, you do not need to make a special package for that. Instead, use pip install pcleaner-cli --no-dependencies. This will just make pip skip installing dependencies. Then just make sure your handpicked list is installed in whatever order to make it work.

Thus, pcleaner-cli is necessary because we need to excise the gui code in its entirety to safely run without pyside, but a pcleaner-cli-colab that simply has less dependencies is not sensible. --no-deps and a custom requirements list take care of that much better.

As for the following points.

Not sure if that means adding any code inside the existing cli codepaths. Extra visualizations are best kept separate so it's clear where they're being used.
- in my cli patch the tr function just acts as a passthrough, like you did I think. It is possible to support a second translation system, but that's honestly just retarded in so many ways, I won't stand for that. If you want translations, the CLI will have to fully depend on PySide6 again, making the existence of pcleaner-cli redundant.
- Not sure what's up with that.
To 4. I don't know anything about that stuff. If you're invested in gradio, I will see if I can accommodate in any way.

Yeah, Panel Cleaner is a tool for primarily cleaning and uses deep learning at the core. It's already multi platform and offers a cli and gui, not sure what sector exactly you wish to expand to, other than notebooks? I won't do web development so that's kinda where it ends for me at least.

BTW, I plan to include manga-ocr in the testbed soon for comparison purposes. I'll check for any dependency collisions at that time.

Also, could you spare a few minutes to help create the ground truth for one or two of the Japanese pages in the testbed set? I'm not quite sure how to start with that.

Also, I'm no expert in manga as your are, maybe we should include more representative pages than the ones I selected in a hurry.

I'm honestly no expert either and don't know any Japanese, but I know a guy who is and I know another guy who can translate. I'll arrange something in the coming days.

Very impressive what you're making.

If you wish to limit dependency overlap, you do not need to make a special package for that. Instead, use pip install pcleaner-cli --no-dependencies.

That settles the issue, then. I concur; it's a sensible solution. I should have looked a bit deeper into pip and Python packages, but until now, I hadn't had the need.

Not sure if that means adding any code inside the existing CLI codepaths. Extra visualizations are best kept separate so it's clear where they're being used.

Absolutely, I continue to believe the MVC paradigm is as relevant today as it was when SmallTalk popularized it. I meant adding additional code in the notebooks. These levels refer to progressively complicating the notebooks. If someday this PR goes mainstream, the notebooks can reuse code from the testbed. If not, the notebooks will remain isolated, utilizing pcleaner's main functionality and adding new code as deemed necessary by the notebook developer.

In my CLI patch, the tr function just acts as a passthrough, like you did, I think. It is possible to support a second translation system, but that's honestly just retarded in so many ways, I won't stand for that. If you want translations, the CLI will have to fully depend on PySide6 again, making the existence of pcleaner-cli redundant.

I was suggesting using a different localization system, not having two of them. But I'm no expert; it's been many years since I built my last cross-platform or native app (thank the gods, what a can of worms!). Localization or GUIs in general is not a field I'm interested in. If you agree to keep English as the only language of the CLI (when installed with pcleaner-cli), that works perfectly for me.

Dependency Management: Investigate the manga-ocr requirement to see why they have pinned the 10.x version of PIL. It's hard to believe they're using some functionality that wasn't in the 9.x release. The Colab team is slow, but they'll probably upgrade PIL (and Python 10) soon.

Your answer has provided a ready solution. In the Idefics notebook, I only install Idefics, not manga-ocr or Tesseract. In the notebook that needs manga-ocr, I will install pcleaner-cli without the manga-ocr dependency. Then, I will install manga-ocr separately without dependencies, and last install the rest of pcleaner + manga-ocr dependencies separately. Essentially, it would be the same solution I'm using now, but with the neat pcleaner-cli instead of the git hack.

To 4. I don't know anything about that stuff. If you're invested in Gradio, I will see if I can accommodate in any way.

No worries, I'm not particularly interested in Gradio; there's little fun there for me. I have to learn about it because nowadays, almost all work in AI is showcased with Gradio. Almost every new paper, research, or new DL-based product has an associated HuggingFace Space with Gradio, even the big players like Google, Meta, OpenAI, and soon Apple.

As with notebooks, supporting it should not involve changing anything in pcleaner main—apart from the minor points we are now contemplating. If you want to extend the audience of PanelCleaner, having both a CLI and a GUI is fantastic. Adding notebooks and a HuggingFace Space could be also a nice touch, providing new users additional avenues to discover, try out, and evaluate PanelCleaner. I won't be the one to do it, though; I'll leave that to others.

How to proceed? I would open a new PR and explore four levels of integration.

And finally, the main point of my comment (I apologize for digressing earlier):

Do you want a new PR with just the bare Colab/Notebook support? It will include the pcleaner-cli, maybe some of the minor changes in main we've discussed (like tr()), touching docs/README, and a couple of demo notebooks. Nothing of importance or that can break main, and quickly done.
Or should we leave it here until and if the testbed stuff is deemed worthy of joining ranks with pcleaner main?

If you're not interested, that's fine, I'd prefer to spend my time on more enjoyable DL/AI stuff along the lines of this PR.

I have now published the pcleaner-cli (it's on pypi, so pip install pcleaner-cli will work). I've addressed the problems that caused in a merge request on your fork https://github.com/civvic/PanelCleaner/pull/1 (necessary due to rebase). I hope that doesn't cause further issues.

You can continue working on building out the notebook side, if you prefer. No need to split off the foundation just for the sake of it.

As for tr the thought of replacing Qt's translation system with something else never even crossed my mind. To do that in a multilingual Qt GUI would be an undertaking of immense foolishness, since support for it is baked into the toolkit and its auto-code-generation tools that I heavily use for the GUI. Not to mention adding a new dependency, and somehow migrating the current translations for German and Bulgarian from Qt's bespoke XML format to some new file format.

I just realized that python 3.10 doesn't support StrEnum, I figured this was for colab running python 3.9 or so.

Turns out there is already a compatibility package for this exact purpose, so the ugly hacks to remove StrEnum aren't necessary.

Please revert "Downgrade to Python 10 as Colab not yet support" 11 a2d15f4f3996378a33b4943547a6a0063c5bea7a and rebase onto master. Doing it in that order should cause no conflicts, optimally.

I took care of it, sorry for wasting your time with https://github.com/civvic/PanelCleaner/pull/1 that was stupid of me.

Let me know if the force push broke anything. A rebase is generally better, but not always when collaborating. Be sure to backup your code before fetching.

I took care of it, sorry for wasting your time with civvic#1 that was stupid of me.

Let me know if the force push broke anything. A rebase is generally better, but not always when collaborating. Be sure to backup your code before fetching.

I'm now officially befuddled and bewildered 🤪 'cause I don't exactly know what just happened, but all is good in the world and seems to roll ok 👏

I'm adding pip and git to the ever-growing list of things I got to look into a lot more

Support Python3.10 again didn't patch structures.py, will fail for those running 3.10

Good to hear that it works. The rebase is a maneuver to pretend you split off your branch from a different point than you actually did, which adds or removes commits from the base branch into your branch.

Whereas the stupid thing I blindly followed github's directions into doing was to just contribute to your fork, instead of contributing to this branch right here in my own project that you opened.

Git is very powerful, it's worth knowing how to do a git rebase properly.

Looks like I need to do it all again then. No clue how that slipped past a global search, but I'll fix it. I don't have python 3.10 anymore, maybe I should get it back for testing.

I did the lazy thing and used a merge commit this time. It should be ok. All fixed now.

Seeing as any more commits to master will be a lot of effort to rebase through with the conflicts I've caused today, it may be wise to take the time to put the notebook foundations in place before building even more on top, just to keep git manageable. That isn't to say it needs to be done right now, I'll leave that up to you.

Before that though, I'd like to know if it's possible to move the _testbed folder out of the pcleaner directory, since it isn't actually part of the package that gets distributed. What was the original issue that necessitated this placement? I suspect it may have been fixed with the introduction of the pcleaner-cli package, but that's just a hunch.

Also, I think it'd still be better to rename it to testbed, without hiding it with _ . It will make sure it appears where it should when sorting alphabetically and not scare people away by having a name that implies something hidden, internal, or private.

VoxelCubes / PanelCleaner

Testbed #94

Introduction of a Possible PanelCleaner Testbed

Overview

Key Features

Impact

Future Directions

Future Work (No particular order or schedule):