search

TensorStack-AI / OnnxStack

C# Stable Diffusion using ONNX Runtime
Apache License 2.0
221 stars 33 forks source link

WebUI, generated image saved with incorrect path. #2

Closed tonual closed 1 year ago

tonual commented 1 year ago

Instead of:

wwwroot\
   images\
     results\
            Output-xkxao1og.png

The image is saved like this: 

wwwroot\
   images\results\Output-xkxao1og.png

(Image name consists of image path)

saddam213 commented 1 year ago

Is this on Mac?

I am unable to replicate this issue as windows does not allow \ in file names.

If you are able to fix this on mac and do a pull request that would be awesome.

Sorry I have no ability to test mac stuff :(

saddam213 commented 1 year ago

Think I may have found the issue, had some pretty average code creating the result filepath that may not have been cross-platform friendly

Commit caf8917 should fix the issue if you could please test

tonual commented 1 year ago

Thanks for the fix, img2txt seems to work just fine now.

There are issues with img2img:

The generated JSON blueprint has an incorrect image path (repeated path) "InputImage": { "ImagePath": "/images/results//images/results/Upload-1alm4noj.png" }, I was looking for a fix tweaking the StableDiffusionHub.cs:

commit

After applying the fix the JSON blueprint looks good, and there are no errors in the console regarding incorrect image file path.

However, once the "Generate" is submitted I got this error:

info: OnnxStack.Web.Hubs.StableDiffusionHub[0] [OnConnectedAsync], Id: lnRLzIy0qtSrlPRWeerW6g info: OnnxStack.Web.Hubs.StableDiffusionHub[0] [ExecuteImageToImage] - New request received, Connection: lnRLzIy0qtSrlPRWeerW6g Scheduler: LMS, Size: 512x512, Seed: 1998097378, Steps: 15, Guidance: 7.5 Prompt - Length: 24, Tokens: 7 N-Prompt - Length: 0, Tokens: 0 fail: OnnxStack.Web.Hubs.StableDiffusionHub[0] [RunStableDiffusion] - Error generating image, Connection: lnRLzIy0qtSrlPRWeerW6g System.AggregateException: One or more errors occurred. ([ErrorCode:InvalidArgument] Input name: 'sample' is not in the metadata) ---> Microsoft.ML.OnnxRuntime.OnnxRuntimeException: [ErrorCode:InvalidArgument] Input name: 'sample' is not in the metadata at Microsoft.ML.OnnxRuntime.InferenceSession.LookupInputMetadata(String nodeName) at Microsoft.ML.OnnxRuntime.InferenceSession.LookupUtf8Names[T](IReadOnlyCollection1 values, NameExtractor1 nameExtractor, MetadataLookup metaLookup) at Microsoft.ML.OnnxRuntime.InferenceSession.Run(IReadOnlyCollection1 inputs, IReadOnlyCollection1 outputNames, RunOptions options) at Microsoft.ML.OnnxRuntime.InferenceSession.Run(IReadOnlyCollection1 inputs, IReadOnlyCollection1 outputNames) at Microsoft.ML.OnnxRuntime.InferenceSession.Run(IReadOnlyCollection1 inputs) at OnnxStack.Core.Services.OnnxModelService.RunInternal(OnnxModelType modelType, IReadOnlyCollection1 inputs) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.Core/Services/OnnxModelService.cs:line 92 at OnnxStack.Core.Services.OnnxModelService.RunInference(OnnxModelType modelType, IReadOnlyCollection1 inputs) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.Core/Services/OnnxModelService.cs:line 68 at OnnxStack.StableDiffusion.Services.SchedulerService.PrepareLatents(PromptOptions prompt, SchedulerOptions options, IScheduler scheduler, IReadOnlyList1 timesteps) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.StableDiffusion/Services/SchedulerService.cs:line 132 at OnnxStack.StableDiffusion.Services.SchedulerService.RunAsync(PromptOptions promptOptions, SchedulerOptions schedulerOptions, Action2 progress, CancellationToken cancellationToken) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.StableDiffusion/Services/SchedulerService.cs:line 62 at OnnxStack.StableDiffusion.Services.StableDiffusionService.GenerateAsync(PromptOptions prompt, SchedulerOptions options, Action2 progressCallback, CancellationToken cancellationToken) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.StableDiffusion/Services/StableDiffusionService.cs:line 42 --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at OnnxStack.StableDiffusion.Services.StableDiffusionService.<>c.<GenerateAsImageAsync>b__3_0(Task1 t) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.StableDiffusion/Services/StableDiffusionService.cs:line 57 at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at OnnxStack.StableDiffusion.Services.StableDiffusionService.GenerateAsImageAsync(PromptOptions prompt, SchedulerOptions options, Action2 progressCallback, CancellationToken cancellationToken) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.StableDiffusion/Services/StableDiffusionService.cs:line 56 at OnnxStack.Web.Hubs.StableDiffusionHub.RunStableDiffusion(PromptOptions promptOptions, SchedulerOptions schedulerOptions, String outputImage, CancellationToken cancellationToken) in /Users/rafalkrzysztof/Documents/repos/OnnxStack/OnnxStack.WebUI/Hubs/StableDiffusionHub.cs:line 196 ^Cinfo: Microsoft.Hosting.Lifetime[0] Application is shutting down...

At this point, I am not sure where to look for the problem. I am guessing the ONNX runtime is looking for (image) sample in metadata and it is probably related again to the image path.
Please advise.

(btw. I totally admire your coding talent and experience with an astonishing number of frameworks, libraries, and design patterns, not even mention the knowledge required to follow StableDifussion algorithm and math)

saddam213 commented 1 year ago

So the problem seems to be with the way the paths are contatinated, its not an issue under windows so I missed it.

If the fix I did earler for Txt2Img works then I should be able to tidy up everywhere I concatinate file paths. this weekend :)

The error you have above is because it cannot find the uploaded image, due to the incorrect filepath, after Generate is clicked the backend will try turn the Url filename into a physical path. that seems to be the issue above, should be a simle fix, however I wont know untill you test as I have no mac

I will let you know ehen I have it fixed up and ready to test

The WebUI is still very new, put it together in a day last week, so there may be a few more issue around :)

Edit: The Bluprint file is not used for anything yet, added it because so many thimes I have made great images and forgot the seed or prompt, the blueprint idea was just a nice way to save all setting that made an output image, it also looks like its handy for debugging.

saddam213 commented 1 year ago

So the first thing we need to do is find were the incorrect path is coming from.

  1. When upload image is done, it will upload an image in base64 format to the FileService UploadImageFile
  2. FileService will save the file to disk and create the FileUrl and return it to the client ImageToImage.cshtml line 368
  3. that FileUrl is added to the Generate request and is sent to StableDiffusionHub OnExecuteImageToImage
  4. Copy the Uploaded file. from Upload-xxxxx to 'Input-xxxxx' (line 122)
  5. Set the full filepath of the image and send to the StableDiffusionService (line 136)

We need to find out which one of these places is concatenating the path wrong on OSX

Also which ONNX model are you using, just to make sure its not a model specific thing and unreated to paths

saddam213 commented 1 year ago

Added a commit that will fix up the blueprint path, but that path does not seem to be uses so not sure thats the problem yet.

This line in the exception is making me think we are using diffenet models perhaps and the input args are not the same Input name: 'sample' is not in the metadata

Will be an easy fix if thats the case

tonual commented 1 year ago

Regarding models all onnx files (unet weights.pb included) I took from https://huggingface.co/runwayml/stable-diffusion-v1-5/tree/onnx

except cliptokenizer.onnx it came from https://github.com/cassiebreviu/StableDiffusion/blob/main/StableDiffusion.ML.OnnxRuntime/cliptokenizer.onnx

Is that collection the same for you?

saddam213 commented 1 year ago

yeah same as me, maybe double check the VaeEncoder path?

I have made some changes regarding how images are used, perhaps it will work now?

tonual commented 1 year ago

It works great now, all good and tasty, img2img as well. Regarding the last issue with

Input name: 'sample' is not in the metadata

I' am afraid it was my issue alone, I must have mistaken vae enc with vae dec. Thank you for the hint on this.