Open iG8R opened 1 month ago
BinitDOX
commented
1 month ago Thanks for bringing this up, I did not even notice it before!
I've now added scaling options:
Usage: python app-stream.py --resize_method=basic
iG8R
commented
1 month ago @BinitDOX
Thank you very much!
Here is the solution for upscaling images using the RealESRGAN_x4plus_anime_6B model after colorization: https://github.com/xiaogdgenuine/Manga-Colorization-FJ
Perhaps the RealESRGAN_x2plus model could be used for this purpose also.
iG8R
commented
1 month ago Due to the presence of CUDA, this fork works extremely fast.
iG8R
commented
1 month ago @BinitDOX Unfortunately, the new fix doesn't work. The following error arises:
(venv) f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main>app-stream.py --resize_method=basic
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\colorizator.py:27: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
self.colorizer.generator.load_state_dict(torch.load(generator_path, map_location = device))
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\denoising\denoiser.py:42: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state_dict = torch.load(weights_path, map_location=torch.device('cpu'))
* Serving Flask app 'app-stream'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
* Running on https://127.0.0.1:5000
* Running on https://192.168.0.3:5000
Press CTRL+C to quit
127.0.0.1 - - [27/Jul/2024 23:49:50] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [27/Jul/2024 23:49:52] "GET /favicon.ico HTTP/1.1" 404 -
127.0.0.1 - - [27/Jul/2024 23:50:05] "OPTIONS /colorize-image-data HTTP/1.1" 200 -
Image Height: None, Image Width: 992
Requested 09a37cb0-c8d0-4d32-9b3c-3f5a9965f8f2 size 992
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\denoising\functions.py:40: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\tensor\python_tensor.cpp:80.)
downsampledfeatures = torch.cuda.FloatTensor(N, Cout, Hout, Wout).fill_(0)
Colorized size 576 in 1.3581924438476562 seconds.
[2024-07-27 23:50:06,768] ERROR in app: Exception on /colorize-image-data [POST]
Traceback (most recent call last):
File "f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\flask_cors\extension.py", line 178, in wrapped_function
return cors_after_request(app.make_response(f(*args, **kwargs)))
^^^^^^^^^^^^^^^^^^
File "f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\flask\app.py", line 865, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\app-stream.py", line 64, in colorize_image_data
color_image_data64 = colorize_image(image, colorizer, colored_img_size, denoiser, denoiser_sigma, (img_width, img_height))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\app-stream.py", line 118, in colorize_image
colorized_img = upscale_basic(colorized_img, original_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\app-stream.py", line 157, in upscale_basic
upscaled_image = cv2.resize(image, target_size, interpolation=cv2.INTER_CUBIC)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.10.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:4155: error: (-215:Assertion failed) inv_scale_x > 0 in function 'cv::resize'
127.0.0.1 - - [27/Jul/2024 23:50:06] "POST /colorize-image-data HTTP/1.1" 500 -Created a new backend, with the ESRGAN super resolution and unexpectedly time taken to colorize at 576px and upscale using super resolution is much less than just coloring at larger size. The quality is better than the original. Thank you for referencing that model!
Re-clone the repository, download and place the generator weights in the networks folder of Backend-New. Then simply run: python app-stream.py (The upscale_method is now set to 'super_resolution' by default)
That error is because I updated the extension (frontend) files, you can fix it by re-loading the extension package into the browser.
BinitDOX
commented
1 month ago This repository desperately needs reorganization and redocumentation.
iG8R
commented
1 month ago Thank you very much for your efforts!
But something went wrong, and there is no reaction from the side of the server when I open a manga page, e.g. the one mentioned above.
Although, when I test whether the addon in Firefox works well in the conjunction with the server, I get that everything is fine, as you can see from the log lines 127.0.0.1 - - [28/Jul/2024 14:28:30] "GET / HTTP/1.1" 200 -.
Here is the addon:
managa-colorizer-anysite-0.3.0.xpi.txt
PS. BTW, why are there so many warnings in the log?
(venv) f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main>app-stream.py
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\colorizator.py:22: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state_dict = torch.load(config.colorizer_path, map_location=self.device)
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\denoising\denoiser.py:42: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state_dict = torch.load(weights_path, map_location=torch.device('cpu'))
F:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\upscalator.py:19: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
model = torch.load(config.upscaler_path, map_location=self.device)
f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\torch\serialization.py:1189: SourceChangeWarning: source code of class 'torch.nn.modules.conv.Conv2d' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\torch\serialization.py:1189: SourceChangeWarning: source code of class 'torch.nn.modules.container.Sequential' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
f:\Manga-Colorization-FJ\Manga-Colorization-FJ-main\venv\Lib\site-packages\torch\serialization.py:1189: SourceChangeWarning: source code of class 'torch.nn.modules.activation.LeakyReLU' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
warnings.warn(msg, SourceChangeWarning)
* Serving Flask app 'app-stream'
* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on all addresses (0.0.0.0)
* Running on https://127.0.0.1:5000
* Running on https://192.168.0.3:5000
Press CTRL+C to quit
127.0.0.1 - - [28/Jul/2024 14:25:33] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [28/Jul/2024 14:26:55] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [28/Jul/2024 14:28:30] "GET / HTTP/1.1" 200 -Oh my bad, I forgot a comma in the Firefox-AnySite extension script. Please update (git pull) and reload the package.
You can ignore those warnings. The first few security warnings occur if we load the entire model (model network code + weights), can be fixed by loading the network first (state_dict), then loading the weights. The source code warnings occur if we essetially load old model with new (updated torch) code, can be fixed by resaving your own copy of the model, then you can always load this saved model.
iG8R
commented
1 month ago Still the issue continues to exist :( The same thing happens in Firefox and Chrome. It seems that the addon does not work at all while the site is loading. I even reinstalled Manga-Colorizer from scratch, but with the same result.
MangaColorizer-server-connection-issue-Video_2024-07-28_17-23-03.webm
iG8R
commented
1 month ago In Chrome:
iG8R
commented
1 month ago Sorry, my bad, it turns out that I had to click the "Colorize!" button to start colorization on the site, although on earlier versions I had colorization automatically started right after loading the page with manga.
iG8R
commented
1 month ago I noticed that 2 files (upscaled.png and colored.png) appear in the root directory while the colorizer is running.
Is it possible to create a cache and at the same time a kind of catalog of the mangas that were already colored, saving the images themselves in a structured way? This way, the unnecessary repeated work by the colorizer can be avoided in the future when visiting these same pages and at the same time store them locally so that users can view them through third-party readers.
Also, it would be great to store finished colore images in the WEBP format, while preserving free disk space.
iG8R
commented
1 month ago And one more thing, can it be possible to make a "Show Original" button in addition to the "Colorize!" button? Since colorization does not always go well with some types of text.
https://senkuro.com/manga/re-monster/chapters/119429828913939988/pages/4
BinitDOX
commented
1 month ago iG8R
commented
1 month ago Thanks a lot again!
BinitDOX
commented
1 month ago The colorizer simply queries the img tags and replaces the grey images with colored. A simple way to make folder structure is to make it exactly like the URL: https://senkuro.com/manga/re-monster/chapters/119429828913939988/pages/4, so senkuro->manga->re-monster->chapters->119429828913939988->pages->4 But if the URL is like: https://mangadex.org/chapter/d72a3eb8-2b20-4c39-a46f-d282eb05e64f/3, there is no manga title. Another way is to give the option of setting the title and chapter in the extension manually. I'll look into this tomorrow.
iG8R
commented
1 month ago Thank you for the explanation. BTW, why do the colors change so weird when the image size differs from 576? As can be seen from the very first project of the original author, there is no such binding to size anywhere in the code at all. https://github.com/qweasdd/manga-colorization
iG8R
commented
1 month ago But if the URL is like: https://mangadex.org/chapter/d72a3eb8-2b20-4c39-a46f-d282eb05e64f/3, there is no manga title. Another way is to give the option of setting the title and chapter in the extension manually. I'll look into this tomorrow.
Besides manually setting the title and chapter, perhaps it's worth considering automatically generating them based on the browser tab's title?
BinitDOX
commented
1 month ago Colors are weird, probably because the model was trained by resizing the images down to 576. Then multiple Conv2D encoder blocks encodes the image down to something like 4 or 2. Then some processing (pattern learn / coloring etc) happens at this latent encoded space, then the image is decoded again by TransposeConv2D decoders back to 576. If you give the model an image size greater than 576 and if it dynamically doesnt add more Conv2D encoders, the encoded latent image stops at a higher dimention. For example if 576 goes down to 4. Then 1152 will go down only to 8. Processing at a larger size will not only take much more time but also return bad results. This could be the reason for bad colors.
I have previously experimented with GANs before (~3y ago), if you want to do it too:
For example in this code: If I pass a 256x256x3 to it, every 'down' block downsizes the image by 2 but also increases the channels. So at down7, image may be 2x2x512, then at the end of final_up, image may be back to 256x256x3. If you pass a larger image, model will accept but the bottleneck will be larger and inefficient.
class Generator(nn.Module):
def __init__(self, in_channels=35, features=64):
super(Generator, self).__init__()
self.down1 = UNetEncoder(in_channels, features, normalize=False)
self.down2 = UNetEncoder(features, features*2)
self.down3 = UNetEncoder(features*2, features*4)
self.down4 = UNetEncoder(features*4, features*8)
self.down5 = UNetEncoder(features*8, features*8, dropout=True)
self.down6 = UNetEncoder(features*8, features*8, dropout=True)
self.down7 = UNetEncoder(features*8, features*8, dropout=True)
self.bottleneck = UNetEncoder(features*8, features*8, normalize=False, dropout=True)
self.up1 = UNetDecoder(features*8, features*8, dropout=True)
self.up2 = UNetDecoder(features*8*2, features*8, dropout=True)
self.up3 = UNetDecoder(features*8*2, features*8, dropout=True)
self.up4 = UNetDecoder(features*8*2, features*8)
self.up5 = UNetDecoder(features*8*2, features*4)
self.up6 = UNetDecoder(features*4*2, features*2)
self.up7 = UNetDecoder(features*2*2, features)
self.final_up = nn.Sequential(nn.ConvTranspose2d(features*2, 3, 4, 2, 1), nn.Tanh())I'll consider the automatic structure generation.
iG8R
commented
1 month ago Oh, thank you so much for such a detailed explanation! I really appreciate it!
iG8R
commented
1 month ago I conducted the following experiment. I resized the image to a width of 1152 pixels, divided it into 4 parts so that the width of the separated images was 576 pixels, and then tried to process them...
Original images:
III
III
Processed images:
III
III
Eh, the color distortion is still there with even worse results.
BinitDOX
commented
1 month ago Will check this soon.
BinitDOX
commented
1 month ago This tiling is another method which could've been used instead of just upscaling., Find closest width divisible by 32, then divide the image into tiles of 576 width, then colorize, then reassemble. But since the colorizer was trained with a dataset distribution containing full images where it probably learnt about context (grass, trees, buildings, relations, etc), if you give it tiled images (different distribution), it fails. That's probably why the other fork, even though it has the code to use tiling, doesn't use it for colorizer but only for super resolution.
But the tiles you provided are rich with context and should've worked.
iG8R
commented
1 month ago These eye-popping colors are just terrible, it's better as it was originally, and if the text is poorly visible, then look at the black-and-white original.
Or maybe there are some parameters that could be passed to the model to correct the color palette, for example, I saw that in the source code of the v1 project there is a parameter sigma, not denoise sigma, but in the v2 project it disappeared somewhere.
BinitDOX
commented
1 month ago I'm unable to find the sigma, can you please link it? You can also play with the denoiser sigma, since it changes the image a bit, you can get different outputs.
The caching configuration will have to be set for each site, although it's quite easy, it has to be hardcoded once manually. I've added it for these sites, you can update and test.
const siteConfigurations = {
'mangadex.org': {
titleSelector: 'a[data-v-5d3b2210]',
chapterSelector: 'div#chapter-selector span[data-v-d2fabe5b]',
getTitle: (document) => document.querySelector(siteConfigurations['mangadex.org'].titleSelector)?.innerText,
getChapter: (document) => document.querySelector(siteConfigurations['mangadex.org'].chapterSelector)?.innerText,
},
'senkuro.com': {
titleSelector: 'p.nav-reader-caption__desktop',
chapterSelector: 'p.nav-reader-caption__mobile',
getTitle: (document) => document.querySelector(siteConfigurations['senkuro.com'].titleSelector)?.innerText,
getChapter: (document) => document.querySelector(siteConfigurations['senkuro.com'].chapterSelector)?.innerText,
},
'chapmanganelo.com': {
titleSelector: 'div.panel-breadcrumb a[title]',
chapterSelector: 'div.panel-breadcrumb a[title]',
getTitle: (document) => document.querySelectorAll(siteConfigurations['chapmanganelo.com'].titleSelector)?.[1]?.innerText,
getChapter: (document) => document.querySelectorAll(siteConfigurations['chapmanganelo.com'].chapterSelector)?.[2]?.innerText,
},
'fanfox.net' : {
titleSelector: 'p.reader-header-title-1 a[href]',
chapterSelector: 'p.reader-header-title-2',
getTitle: (document) => document.querySelector(siteConfigurations['fanfox.net'].titleSelector)?.innerText,
getChapter: (document) => document.querySelector(siteConfigurations['fanfox.net'].chapterSelector)?.innerText,
},
'mangakakalot.com' : {
titleSelector: 'div.breadcrumb span[itemprop=name]',
chapterSelector: 'div.breadcrumb span[itemprop=name]',
getTitle: (document) => document.querySelectorAll(siteConfigurations['mangakakalot.com'].titleSelector)?.[1]?.innerText,
getChapter: (document) => document.querySelectorAll(siteConfigurations['mangakakalot.com'].chapterSelector)?.[2]?.innerText,
}
};
Provide me a list of sites that you want it to work for. I usually only use a site like mangakakalot.
iG8R
commented
1 month ago I'm unable to find the sigma, can you please link it?
You can also play with the denoiser sigma, since it changes the image a bit, you can get different outputs.
I've already done it, but to no avail. https://github.com/qweasdd/manga-colorization-v2/issues/17#issuecomment-2254712104
The caching configuration will have to be set for each site, although it's quite easy, it has to be hardcoded once manually. I've added it for these sites, you can update and test.
Thank you, it's great! Perhaps it would be more convenient to dedicate a separate configuration file for sites whose data will be cached? My sites' list, offhand from memory: Besides senkuro.com readmanga.live remanga.org manhuatop.org manhuaus.com
BinitDOX
commented
1 month ago Perhaps it would be more convenient to dedicate a separate configuration file for sites whose data will be cached?
Great idea! will do that tomorrow along with the sigma param.
BinitDOX
commented
1 month ago Configuration file created. The previous v1 project also had an option to give color hints, basically driving the model to follow those colors. Since it was manual work, it was probably removed in v2, along with params like sigma, etc.
iG8R
commented
1 month ago It's not that urgent, but is it possible to implement the same feature to give color hints in MC, since the eye-gouging colors in some images can be really upsetting. For example, when someone is depicted eating steak or fish, and they are green, as if they have gone bad. https://mangadex.org/chapter/89ddbe27-e814-4d91-8701-318e6ff7a082/2
vatavian
commented
1 month ago I have also noticed how often food tends to be colored green and wondered whether this is because the training data is skewed toward green food. Does our latest version let us try this page with 576 width to see if that helps? All the other coloring in this one looks great, though.
iG8R
commented
1 month ago @vatavian The example above was colorized with the latest version without changing the output size, i.e. it was colorized at 576 and then 4x upscaled.
BinitDOX
commented
1 month ago let us try this page with 576 width
Since the model works best with 576, it is now fixed in the latest version, you can change and experiment with it ~line 256 in app-stream
when someone is depicted eating steak or fish, and they are green
Most probably a model problem, a denoise of 100 changes it's color, but also removes details from the image.
iG8R
commented
1 month ago Most probably a model problem, a denoise of 100 changes it's color, but also removes details from the image.
I agree, it looks better in relation to food, but the fact that there are practically no details and the picture in general turns out to be blurred and the characters' faces are deformed does not give this method an advantage, because in fact you have to choose between bad and bad.
Right away I want to thank you for this wonderful software that allows to colorize manga on the fly. I've known about the source app for a while now - https://github.com/qweasdd/manga-colorization-v2/, and I'm familiar with its limitation where processing images wider than 576px results in incorrect coloring.
Is it possible to colorize the image at a size most suitable for the program, i.e. x=576px, and only after this, scale it to a larger size?
Let me explain with an example.
https://mangadex.org/chapter/d72a3eb8-2b20-4c39-a46f-d282eb05e64f/3
Original image (1125 x 1600 Pixels)
How does the image look after processing if the program processed it at dimensions of
x = 576px, as originally intended (576 x 820 Pixels)?This is what the image looks like when processed at sizes exceeding 576px in width (960 x 1367 pixels).