search

imxieyi / waifu2x-ios

iOS Core ML implementation of waifu2x
MIT License
551 stars 58 forks source link

Image decay on the new A12 Bionic chip #8

Closed imxieyi closed 4 years ago

imxieyi commented 6 years ago

I have received several reports that the image quality will decay on iPhone XS. The following is an example (pay attention to the color gradient): Before processing: image1 After processing: image2 Result on iPhone 8 for comparison: image3 The issue should come from the new neural engine on A12 chip. If this can be confirmed, neural engine should be disabled in this app. For details, it might because of one of the following reasons:

  1. Float16 numbers are cast to Float8 or lower precision before sending to neural engine.
  2. The model is somehow "optimized" by Core ML framework for better performance on the neural engine.

If this cannot be solved, I will consider rewriting the whole library in MPSNN which can only run on the GPU. It will also significantly increase the size of the app.

Since I don't have any iPhone XS/XS Max, I cannot debug this problem for now. Thanks ahead for anyone who can help investigate into this issue!

sfslowfoodie commented 5 years ago

I can confirm the image quality issue on my XS 256GB. Artifacts are visible on color gradients, the final images are unusable. Running in CPU only mode quality is good as expected, however the XS takes about twice as much time to complete a task when compared to a 7 Plus. I am available for trying out new builds.

imxieyi commented 5 years ago

Turned out that setting computeUnits in MLModelConfiguration to cpuAndGPU can bypass neural engine for now.

The MPSNN implementation is partially finished. It will be released if it can be faster or results in better quality than Core ML in benchmarks.

In the future if I could get hands on a device with A12 chip, I still wish to try fixing for neural engine since it should be much faster even than GPU mode.

sfslowfoodie commented 5 years ago

With regard to the waifu2x app in the App Store (which is what I am testing), is/will this app rely on both cpu And GPU to run the Core ML model, then?

imxieyi commented 5 years ago

With regard to the waifu2x app in the App Store (which is what I am testing), is/will this app rely on both cpu And GPU to run the Core ML model, then?

Currently it only has Core ML as back end, which relys on GPU. If GPU is not supported it will fallback to CPU.

sfslowfoodie commented 5 years ago

What bugs me (what I don’t understand since I am not an expert on this matter) is why the performance on iPhone 7 Plus (A10 SOC, no neural engine) is still ~2X faster than on XS, excluding neural engine factors. Running both iPhone models in Cpu Only mode, I see the A12 benefits (~35% faster cpu than the A10 cpu). Disabling the Cpu Only however, the latest XS app still generates artifacts, implying it is still trying to use the neural engine, instead of relying on the GPU (as the A10 does).

imxieyi commented 5 years ago

Weird... The model.configuration.computeUnits should have been set to .cpuAndGPU. It seems that this fix is ineffective. I will try to get hands on a device with A12 chip. Also a MPSNN update will be considered before then.

ZhongDev commented 5 years ago

Can confirm, also an issue on A13

imxieyi commented 5 years ago

Can confirm, also an issue on A13

Can you provide a demo image and configurations? I cannot reproduce this issue on iPhone 11 Pro even with neural engine enabled.

ZhongDev commented 5 years ago

Sorry, I meant with coreML implementation. There is still some banding; not as much as a12, but not as smooth as MPSCNN.

Overall it’s fine but I’ll see if I can get an example once I get home from school

sfslowfoodie commented 5 years ago

My observations comparing MPSCNN to Core ML on iPhone XS: while under iOS 12.x the artifacts generated by the Core ML version were unbearable, under iOS 13.x (13.1 (17A5844a)) the results are much better! Outputs: 4X, High Core ML (3.5s) MPSCNN (7.8s)

MPSCNN is slightly better, but iOS 13.x improves a lot the Core ML implementation on the A12 SOC which is at least 2X faster.

Cheers!

On Wed, Sep 25, 2019 at 22:47 Jason Zhong notifications@github.com wrote:

Sorry, I meant with coreML implementation. There is still some banding; not as much as a12, but not as smooth as MPSCNN.

Overall it’s fine but I’ll see if I can get an example once I get home from school

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/imxieyi/waifu2x-ios/issues/8?email_source=notifications&email_token=ALBHMYX7KG4MEWG3MXAV4STQLPE7JA5CNFSM4F3OXAOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7TLRDI#issuecomment-535214221, or mute the thread https://github.com/notifications/unsubscribe-auth/ALBHMYSE7LBQTTXQVEGQNF3QLPE7JANCNFSM4F3OXAOA .