Added parallel mode to utilise power of all CPUs

JayXon / Leanify

lightweight lossless file minifier/optimizer

MIT License

831 stars 75 forks source link

Added parallel mode to utilise power of all CPUs #67

Closed doterax closed 3 years ago

doterax commented 3 years ago

Taskflow added from sources to make AppVeyor happy Changed c++14 to c++17 to support taskflow Added option -p --parallel to enable parallel mode Logs changed to prevent mess in it

doterax commented 3 years ago

Need help with building these configurations: Clang-Release; Platform: x86, Clang-Release; Platform: x64 AppVeyor build failed with error: cannot use 'throw' with exceptions disabled

doterax commented 3 years ago

Thank you for review!

I processed all your requests. Except removing not needed files from taskflow. That because I have some plans for them.

Looks like problems with linking in Clang configurations. I will check them tomorrow on a different machine. Or if you have suggestions don't hesitate to contact me.

Cheers :)

doterax commented 3 years ago

Interesting, do you mind sharing the plans?

Yes, sorry.

My future plans is:

change the way how comand line arguments processed
add more command line options for zopflipng (for example diffirentiate number of iterations for small and big files for zopfli)
try playing with guetzli-cuda-opencl for JPEG (https://github.com/ianhuang-777/guetzli-cuda-opencl/tree/opencl)

doterax commented 3 years ago

Travis CI build failed due to not full support of C++17.

https://stackoverflow.com/questions/40177006/c-variant-no-such-file-or-directory

JayXon commented 3 years ago

try playing with guetzli-cuda-opencl for JPEG (https://github.com/ianhuang-777/guetzli-cuda-opencl/tree/opencl)

I don't think we can use guetzli in leanify, because leanify is a lossless tool but guetzli is lossy.

Travis CI build failed due to not full support of C++17.

Sorry about this, I've already switched from travis ci to github actions, it should show up if you add a new commit.

doterax commented 3 years ago

Could you address this? I don't see any new commits

Yes, sorry I set up system to reproduce Clang linking issues and didn't commit before fix. And now AppVeyor works.

I had to disable optimisation for Clang, and this helped. Someone also faced same issue https://github.com/taskflow/taskflow/issues/300

I don't think we can use guetzli in leanify, because leanify is a lossless tool but guetzli is lossy.

Ok, I understand. I removed all unused taksflow files.

I've already switched from travis ci to github actions, it should show up if you add a new commit.

gcc build on ubuntu-16.04 still not fully support C++17

Rest configurations on ubuntu are failed, and for now I have no idea how to fix them. Because problem looks same as in Clang, I tried to remove optimisation from makefile, but this is not worked.

doterax commented 3 years ago

Try adding -pthread to the makefile?

It works. Thank you.

Do we really need to disable all the optimizations?

I'm trying to figure out why it helps and how to preserve all optimizations.

doterax commented 3 years ago

I'm trying to figure out why it helps and how to preserve all optimizations.

I'm working on it. Looks like rare LLVM bug. Have plans to rewrite TLS part of taskflow an enable optimisation for Clang. Also somehow SEH on Windows also involved.

doterax commented 3 years ago

Looks like everything is green now :)

JayXon commented 3 years ago

Thanks!