FMD use High CPU when download manga

[nattapan]

start with 0.9.131 till .133 FMD use much more CPU power to download file , I use this setting before .131 but it didn't use that much of CPU power ,I notice this because of my NB breath out of very hot wind when I download chapters from manga sites

Thanks in advance.

[kmvi]

From which website you are downloading?

[nattapan]

before I download from MangaFox , but right now I switched to MangaRock

[kmvi]

Mangarock uses webp as image format. We have to convert it to PNG. We use official google codec for this. But unfortunately conversion process takes a lot of CPU resources. Other websites don't require image conversion.

[riderkick]

IIRC scanline is slow in fpc. http://wiki.freepascal.org/Fast_direct_pixel_access Try to use TFPCustomImage instead.

Or his download speed is to fast and he download to much to give his computer a break.

[nattapan]

I see , can you have an option to save as webp too ,my viewer support webp too. My download speed is around 10 - 50 Mbps. I suggest that must find some opencl codecs to handle this convert task.

[Tmp341]

I think reducing the first two options might reduce system usage too. I always use 1 for "number of downloaded tasks" and "number of files". It is an habit from my Jdownloader times. It was slowing my internet usage, consuming a lot of ram etc.

[nattapan]

Of course reduce working threads will reduce CPU usage , But in my opinion I chose option to save as webp because my viewer has webp codec ,with this it kill two birds with one stone , 1 is reduce CPU usage ,2 is reduce files size , and I still got speed of my link too. P.S. after researched accelerated OpenCL codec for webp still nowhere found.

[birdmanravo]

Can we have the option to not convert the images to PNG? FMD is made for Windows and Windows can view WEBP images.

[kmvi]

I did a little research and it turned out that the problem actually is in TFPWritePNG, not in the decoder. I looked for how to replace this class. dwebp utility from libwebp distribution uses either WIC or libpng. BGRABitmap also looks promising. I'll try to experiment with these libraries...

[riderkick]

Encoding a png image is pretty straight forward, its lossless. Especially if we don't use palettes. But png can compress the stream using deflate. Maybe that's the problem. IIRC There is a compression level that we can adjust in png writer. I've been reading this unit when writing imp2pdf years ago.

BGRABitmap using the same reader/writer from fpc, with a bit of modification. I've been using BGRABitmap for my various projects for years. Its add about 2mb to binary size. So, I don't use it. We are not drawing anything advance for now.

[riderkick]

@kmvi please test https://github.com/riderkick/FMD/commit/1a19d243df2888389f7da386c67d4933fbffd590 and change it to clfastest if necessary.

[kmvi]

Benchmarking (webp size 1 321 542):

• TFPWriterPNG, default settings, png size 3 141 632

  WebPGetInfo: 0,000023
  Bitmap create: 0,000150
  WebPDecodeBGRAInto: 0,073358
  SaveToStream: 1,014445

• TFPWriterPNG, no compression, png size 4 867 247

  WebPGetInfo: 0,000004
  Bitmap create: 0,000144
  WebPDecodeBGRAInto: 0,073643
  SaveToStream: 0,276625

• TFPWriterPNG, fastest compression, png size 3 338 708

  WebPGetInfo: 0,000004
  Bitmap create: 0,000152
  WebPDecodeBGRAInto: 0,073756
  SaveToStream: 0,811436

  I think clfastest doesn't fit...

[riderkick]

That can be it. 1 second for compressing data. 8 thread with continuous process sure can give an overhead to the machine.

[kmvi]

JPEG benchmarking:

• TFPWriterJPEG, default settings (q = 75), jpg size = 248 535

  WebPGetInfo: 0,000005
  Bitmap create: 0,000041
  WebPDecodeBGRAInto: 0,072659
  SaveToStream: 0,172377

• TFPWriterJPEG, q = 90, jpg size = 408 092

  WebPGetInfo: 0,000004
  Bitmap create: 0,000148
  WebPDecodeBGRAInto: 0,072694
  SaveToStream: 0,185847

[riderkick]

We can make an options in Save to to save webp as webp,jpeg,png. Possibly also the options for compression level for png and jpeg. Store the global variable in FMDVars unit.

[kmvi]

• WIC encoder, png size 2 024 991

  WebPGetInfo: 0,000007
  Bitmap create: 0,000054
  WebPDecodeBGRAInto: 0,074210
  SaveToStream: 0,249836

  WIC PNG encoder gives best results, no additional libs required, but it Windows-only (version >= XP SP3) and I don't know whether it works well in Wine.

[kmvi]

4 threads, 6 chapters.

• compression none - 50 s, 765 MB (!)
• compression fastest - 61 s, 161 MB
• compression default - ~120 s, 138 MB

Despite the high CPU utilization, the fastest mode looks the most reasonable.

[riderkick]

Please test and close if ok.

[kmvi]

It's ok, except that img2pdf cannot read raw webp. zip/cbz also not working unless conversion to png/jpg is set:

13:10:42:567 TTaskThread, failed to compress.

[riderkick]

zip/cbz just miss the webp extension.

[riderkick]

Well I guess we need to write TFPCustomImageHandler for webp and register it in ImgInfos. Just a reader is enough.

[kmvi]

I'll look into it

[riderkick]

After looking again at img2pdf, I think we don't need to write a new handler. Just register a method to read webp stream and size. And add new method in img2pdf to convert webp to png and read as png.

[riderkick]

@kmvi please test

[kmvi]

@riderkick :+1:

[ralvarador]

Then, It will be customizable the way that the files are converted; something like - convert all to JPG using "n Level Compression" - "optimum Huffman" - "progessive"... blah... blah.."? or, is this only for webp?

[riderkick]

Only for webp. Convert to jpg always degrade the quality and maybe increase the size compare to original. Some png file are smaller. I often found indexed png for black and white image. Converting other image doesn't really give much benefit. They usually already optimized for web page, which is optimized for smaller size.