Use real paper scribbles

[woutersf]

Hi, Thanks for creating this. It really sparked the creativity for me and my kids. I could not resist thinking about adding the paper scribbels functionality to this. (if this is not supposed to be a ticket, certainly close it and point me to the right direction.

I have created a fork: Paper sketch diffusion. https://github.com/woutersf/paper-sketch-diffusion

The idea is to allow your input to also allow a picture of a paper sketch.

Therefor I added a button (not styled yet)

Which then promps the camera permission:

And you are allowed to take a picture of a sketch.

I also transform the sketch to black and white and threshold it (black or white contrast) so that it is "scribblable" .

I then want this image data to flow into the scribble (see red arrow above), This is where I'm stuck now. I would like to know how to flow the image data I have and not use the canvas that was hand drawn. You will see in my code that I have no clue about React. https://github.com/woutersf/paper-sketch-diffusion/commit/14a5f477deabc9bf3bc3d2bdf4ec19cee069dd8d

You can easily try it out since you don't even need the replicate key to get this working. Any help is much appreciated.

[zeke]

Hey @woutersf that's pretty neat. I have also had a lot of fun with my kids on this.

Thanks for opening an issue and describing where you're stuck.

Haven't looked at the code yet, but I think at a high level this should work. Your input will create a bitmap image, and that image should then be uploaded using the upload-file module and used as an alternative input, instead of the "Canvas" output.

[woutersf]

Hi Zeke, Thanks for that Tip, i got it working like that.

[woutersf]

This is how it works now:

[woutersf]

I am now wrestling with React to make a PR. 😅

[zeke]

Nice! Keep me posted. I can help out if needed.

[woutersf]

@zeke I've fabricated something here: https://github.com/replicate/scribble-diffusion/pull/35 Any help is certainly appreciated.

[cynepnaxa]

so, how's it going with that lovely feature? i'd prefer simple "upload sketch" button :) i launch it on my pc, but i don't have camera on it. So i tried to use it from my android phone. "Open camera" and "Take photo" doesn't work in chrome(cannot read properties of undefined(reading getUserMedia'')) and firefox(navigator.mediaDevices is undefined).