search

ml5js / ml5-library

Friendly machine learning for the web! 🤖
https://ml5js.org
Other
6.5k stars 901 forks source link

Style transfer on Frame Buffer Object ? #246

Closed b2renger closed 4 years ago

b2renger commented 5 years ago

Hello everyone,

I've played with the style transfer examples and they are pretty fun, thanks for making this accessible !

One thing I would like to do is do some style transfer on a frame buffer object : something like a createGraphics from p5js in which I could build some abstract graphics and see how it goes.

Do you think thinks it is do-able ?

I've looked at the style transfer class it looks deeply entangled with the use of a video element and I would like to be able to use with any king of p5js image / offscreen buffer. I think it would be better to build a whole other class.

Any thoughts on that idea ?

shiffman commented 5 years ago

Yes, we would like to support this! Are you sure it doesn't work? Have you tried applying it to a canvas image? I believe this is already supported but obviously could be wrong here! (This is related to #206)

b2renger commented 5 years ago

Hi @shiffman !

Sorry for the delay, I've posted a little example that doesn't work here : https://github.com/b2renger/ml5-experiments/tree/master/Styletransfer_from_Pgraphics

shiffman commented 5 years ago

Ah, please update your ml5 version in index.html! It should be 0.2.3 now. Note you will also need to use lowercase ml5.styleTransfer() and you should also be able to load the model in preload() which will simplify things!

b2renger commented 5 years ago

Made the changes but I still get the same error. Btw a live version is available here if you want to check : https://b2renger.github.io/ml5-experiments/Styletransfer_from_Pgraphics/

When I log the result I get "undefined" and the error below 

tf-core.esm.js:17 Uncaught (in promise) Error: pixels passed to tf.fromPixels() can not be null
    at e.fromPixels (tf-core.esm.js:17)
    at e.fromPixels (tf-core.esm.js:17)
    at fromPixels_ (tf-core.esm.js:17)
    at Object.fromPixels (tf-core.esm.js:17)
    at t.<anonymous> (index.js:107)
    at x (runtime.js:62)
    at Generator._invoke (runtime.js:296)
    at Generator.e.(/ml5-experiments/Styletransfer_from_Pgraphics/anonymous function) [as next] (https://unpkg.com/ml5@0.2.3/dist/ml5.min.js:106:335809)
    at a (asyncToGenerator.js:17)
    at asyncToGenerator.js:15
e.fromPixels @ tf-core.esm.js:17
e.fromPixels @ tf-core.esm.js:17
fromPixels_ @ tf-core.esm.js:17
fromPixels @ tf-core.esm.js:17
(anonymous) @ index.js:107
x @ runtime.js:62
(anonymous) @ runtime.js:296
e.(anonymous function) @ runtime.js:114
a @ asyncToGenerator.js:17
(anonymous) @ asyncToGenerator.js:15
t @ _export.js:36
(anonymous) @ asyncToGenerator.js:14
(anonymous) @ index.js:128
(anonymous) @ index.js:103
x @ runtime.js:62
(anonymous) @ runtime.js:296
e.(anonymous function) @ runtime.js:114
a @ asyncToGenerator.js:17
(anonymous) @ asyncToGenerator.js:15
t @ _export.js:36
(anonymous) @ asyncToGenerator.js:14
(anonymous) @ index.js:103
transferImages @ sketch.js:42
joeyklee commented 4 years ago

Hi @b2renger! If you don't mind giving this a try with the latest version of ml5 - we're at 0.4.3 now. Style transfer seems to have some issues in firefox, so maybe also testing in Chrome will help.

If this is still an issue, please feel free to add any comments! Thanks!

b2renger commented 4 years ago

@joeyklee thanks for the ping,

I bumped all the versions

But I still do get the same results on chrome or firefox.

It seems that passing a "createGraphics()" object to the ".transfer()" function doesn't work, as tensorflow can't locate the pixel array

joeyklee commented 4 years ago

@b2renger - ah! Interesting. Ok Let's keep this issue open and we can make sure to address this. I will mark this as a bug and we will do our best to figure out the issue.

EmmaGoodliffe commented 4 years ago

I made a quick sketch that demonstrates how you could do this with the current version of ml5 even though I do think this should be implemented in the library itself.

Normal sketch: https://editor.p5js.org/EmmaG/sketches/J8PrwLoiu Fullscreen: https://editor.p5js.org/EmmaG/present/J8PrwLoiu

Edit: I forgot to say that this is a direct fix of @b2renger's question.

EmmaGoodliffe commented 4 years ago

I made a PR for this :)

832

joeyklee commented 4 years ago

@EmmaGoodliffe - Excellent! Thanks for the helpful feedback on this issue. I'll have a look at the PR 🙏