search

livepeer / bounties

Livepeer Software Bounties Portal.
https://livepeer.org
4 stars 1 forks source link

Outpainting Pipeline Implementation on ai-runner Bounty [$850] #56

Open JJassonn69 opened 2 months ago

JJassonn69 commented 2 months ago

Overview

We are excited to announce the expansion of our AI Network with the introduction of Outpainting pipeline. This feature will enable advanced image expansion and modification techniques, improving the versatility of our AI network. 🔥

We are reaching out to the community to contribute to the implementation of this this essential pipeline within the AI-worker side of the AI-Network. Your contribution will be an invaluable addition to our image editing capabilites, pushing Livepeer AI Network at the forefront of AI-media generation. 🚀

Required Skillset

Bounty Requirements

  1. Implementation: Develop a functional /out-paint route and pipeline within the AI-worker repository. This new pipeline should be accessible through Docker on port 8700.
  2. Functionality: The pipeline must be able to take an initial image, and text input as a prompt, return an modified image as the result. It should also allow for optional parameters like negative_prompt, controlnet_conditioning_scale, num_inference_steps, and guidance_scale to be adjustable according to the outpainting model utilized.

Example workflow

image

Originally posted by @stronk-dev in https://github.com/livepeer/bounties/issues/56#issuecomment-2328226172

Scope Exclusions

Implementation Tips

To learn how to create a new pipeline, refer to the Hugging Face pipeline documentation. You can also explore the following pull requests to see how other pipelines were implemented:

In some cases, you may not be able to integrate the new pipeline directly into the regular AI Runner Docker container due to dependency conflicts or missing packages. If this occurs, you can follow the approach outlined in this example to create a custom container for the pipeline. This approach uses the regular AI runner as the base while keeping the base container lean.

Additionally, keep the following best practices in mind:

How to Apply

  1. Express Interest: Comment on this issue to show your interest and explain why you are a great fit for this task.
  2. Await Review: Our team will review expressions of interest and select the most suitable candidate.
  3. Get Assigned: If chosen, the GitHub issue will be assigned to you.
  4. Start Working: Begin the task! If you need help or guidance, comment on the issue or join discussions in the #developer-lounge channel on our Discord server.
  5. Submit Your Work: Create a pull request in the relevant repository and request a review.
  6. Notify Us: Comment on this GitHub issue when your pull request is ready for review.
  7. Receive Your Bounty: Once your pull request is approved, we will arrange the bounty payment.
  8. Earn Recognition: Your valuable contributions will be highlighted in our project's changelog.

We appreciate your interest in contributing to our project! 💛

[!WARNING] Please wait for the issue to be assigned to you before starting work. To prevent duplication of effort, submissions for unassigned issues will not be accepted.

stronk-dev commented 2 months ago

image Not sure if this is a requirement for good outpainting results, but just wanted to drop this image from Reddit here for the person that will be working on this bounty. Apparently generating a mosaic over the area to be repainted helps?

rickstaa commented 2 months ago

image Not sure if this is a requirement for good outpainting results, but just wanted to drop this image from Reddit here for the person that will be working on this bounty. Apparently generating a mosaic over the area to be repainted helps?

@stronk-dev damn that image is great! I added it as an example workflow to the bounty description. Thanks for jumping in 🙏🏻❤️.

lukiod commented 1 month ago

will work on it after finishing the stable-audio pipeline work if someone took on in between no problem will require a time period of a week to finish the previous bounty

aybanda commented 1 month ago

@JJassonn69 can you assign it to me?

rickstaa commented 1 month ago

Hi @aybanda, thanks for your patience! We've been wrapped up with the public launch at Token2049. It’s fantastic to have you contributing to our open-source project 🚀. I’ve assigned you to the bounty now. Don't hesitate to reach out if you have any questions!

aybanda commented 1 month ago

@rickstaa will do my best in delivering this 👍 Thank you

rickstaa commented 3 weeks ago

Hi @aybanda, how's the progress on your bounty implementation? Is there anything we can do to assist?

Also, I came across some exciting news— the Transformers library will add support for a new model called ProPainter, which excels at both inpainting and outpainting. You can check it out here: ProPainter on GitHub.

Let me know if this might be helpful for your work!

aybanda commented 3 weeks ago

Hey @rickstaa Thanks for checking in. I'm making good progress on the bounty implementation, will give you an update very shortly.

aybanda commented 2 weeks ago

Hey @rickstaa

I'm almost ready with the code for the image_outpainting pipeline using the ProPainter model. Here's a summary of my current implementation:

I'm using the "ruffy369/propainter" model from Hugging Face. and handling image resizing and mask creation for outpainting.

I have a few questions:

  1. We're currently using AutoPipelineForInpainting for both inpainting and outpainting tasks, manipulating the input image and mask to achieve outpainting. Is this the most effective approach with ProPainter, or do you recommend any adjustments to better handle outpainting specifically?

  2. Do you have any additional recommendations or best practices for integrating ProPainter into our pipeline, especially for outpainting tasks?

Any insights or suggestions would be greatly appreciated as I prepare to push this implementation. Thank you!