jalagar / animated-art-engine

A generative engine that takes various png layers on a sprite sheet format, combines them and then converts them into a .gif file
MIT License
167 stars 62 forks source link

Welcome to the Generative Animated Engine v3.1.3 🐀

[10 minute read]

This repo used to be called jalagar/Generative_Gif_Engine but because it now supports GIF, MP4, it was renamed to jalagar/animated-art-engine. v3.0.0 is the beginning of the animated era.

Check out this Youtube Tutorial on how it works!

Table of Contents

Summary

This python and node app generates layered-based gifs/MP4 to create animated NFT art! It is faster, simpler, and produces higher quality gifs/MP4s than any other open source animated generative tool out there. It also contains many more features including but not limited to stacking layers, if-then, ETH/Solana/Tezos, preview images, inserting legendaries, gifs/MP4, batching to support hundreds of layers, and multiprocessing.

Export your animation as a png image sequence, organize your layer folders with rarity, and the code does the rest! I plan to actively maintain this repo and enhance it with various tools for months to come so be sure to ask questions in the discussion and write issues.

There are three steps:

  1. [Python] Converts layers into spritesheets using PIL. This step can be skipped if you already have the spritesheets, but is useful if you want to start with png files and makes the artist's life easier!
  2. [Node] Create generative spritesheets from the layers from step 1.
  3. [Python + gifski/ffmpeg] Convert spritesheets to gifs/MP4 using Python and gifski or ffmpeg for MP4.

Checkout this Medium post and How does it work? for more information!

Here's an example final result (or you can download the code and run it and see more bouncing balls :)). It is also pushed to production on OpenSea.

EDIT tool now supports z-index/stacking, grouping, if-then statements, and incompatibilities. See this section for more information. Here is an example of having one layer that is both in front and behind the ball.

Samples

See below for some examples of other artists and devs using this tool for their own collections! Feel free to reach out to me or create a PR with your examples!

@PxlSyl

@Lions_wtf

Dominic Overdrive NFT

Requirements

Install an IDE of your preference. Recomended

Install Node.js v16.14.2. In order to install a specific version of node, you can use nvm. On Mac you can brew install nvm, on Windows you can follow these instructions https://www.freecodecamp.org/news/nvm-for-windows-how-to-download-and-install-node-version-manager-in-windows-10/. Then run nvm install 16.14.2 and nvm use 16.14.2

Install the latest version of Python 3. I am currently using 3.8.1 but anything above 3.6 should work.

If you want to output gifs then:

Install gifski. I recommend using brew brew install gifski if you're on Mac OSX. If you don't have brew you can install it using brew on Mac OSX. Or if you're on Windows you can install it using Chocolatey: choco install gifski.

If you're on Linux, some people were having issues with gifski so you can skip installing it. You will have to set the gifTool config to imageio instead (see later instructions).

If none of those methods work, follow instructions on gifski gifski Github. Gifski is crucial for this tool because it provides the best gif generation out of all the tools I checked out (PIL, imageio, ImageMagic, js libraries).

If you want to output MP4s then:

Install ffmpeg. I recommend using brew brew install ffmpeg if you're on Mac OSX. If you don't have brew you can install it using brew on Mac OSX. Or if you're on Windows you can install it using Chocolatey: choco install ffmpeg.

If you plan on developing on this repository, run pre-commit to install pre-commit hooks.

If you're on Windows you can optionally install Make by running choco install make. Make is already pre-installed on Mac.

Installation

If you have any issues with this command, try running each separate command:

   python3 -m pip install --upgrade Pillow && pip3 install -r requirements.txt

   cd step2_spritesheet_to_generative_sheet && npm i

Each environment can be different, so try Google your issues. I'll add a few known issues below:

Known issues:

How to run?

Load the png or gif files into the /layers folder where each layer is a folder, and each folder contains another attribute folder which contains the individual frames and a rarity percentage. For example if you wanted a background layer you would have /layers/background/blue#20 and /layers/background/red#20.

In each attribute folder, the frames should be named 0.png -> X.png or 0.gif. See code or step 1 for folder structure. The code will handle any number of layers, so you could have a layer with two frames, another layer with one frame, and another with 20 frames, and as long as you pass numberOfFrames = 20, then the layers will be repeated until they hit 20.

EDIT You can leave the frame names whatever you want, and set useFileNumbering to false. This makes it easier if you have hundreds of frames and don't want to rename each one.

Update global_config.json with:

  1. 'totalSupply' : total number of gifs/MP4 to generate.
  2. 'height' : height of one frame. This should be equal to width. Default is 350 (see [https://docs.opensea.io/docs/metadata-standards#:~:text=We%20recommend%20using%20a%20350%20x%20350%20image](OpenSea recommendation))
  3. 'width' : width of one frame. This should be equal to height. Default is 350 (see [https://docs.opensea.io/docs/metadata-standards#:~:text=We%20recommend%20using%20a%20350%20x%20350%20image](OpenSea recommendation))
  4. 'framesPerSecond' : number of frames per second. This will not be exact because PIL takes in integer milliseconds per frame (so 12fps = 83.3ms per frame but rounded to an int = 83ms). This will not be recognizable by the human eye, but worth calling out.
  5. 'numberOfFrames' : number of total frames. For example you could have 24 frames, but you want to render it 12fps.
  6. 'description' : description to be put in the metadata.
  7. 'baseUri' : baseUri to be put in the metadata.
  8. 'layersFolder': this is the folder that you want to use for the layers. The default is layers, but this allows you to have multiple versions of layers and run them side by side. The current repo has four example folders, layers, layers_grouping, layers_if_then, layers_z_index which all demonstrate features from nftchef's repo.
  9. 'quality': quality of the output, 1-100.
  10. 'gifTool': pick which gif generation method to use, gifski or imageio. Gifski is better overall, but some people were having issues with it on Linux. Also imageio will work for more pixel art, so if you don't want to download Gifski you can set this to imageio.
  11. 'MP4Tool': pick which MP4 generation method to use. Only supports ffmpeg at the moment.
  12. 'outputType': select gif or mp4.
  13. 'useBatches': set to true if you want to take advantage of batching. Otherwise does nothing.
  14. 'numFramesPerBatch': number of frames for each batch. See batching for more information. Only does something if useBatches is set to true.
  15. 'loopGif': true if you want to loop the gif, false if you don't want to loop it.
  16. 'useMultiprocessing': true if you want to use multi-processing which will speed up step1 and step3. You can configure how many processors to use with processorCount. Use at your own discretion, I would recommend slowly increase processorCount and monitor CPU usage, this could crash your computer.
  17. 'processorCount': Number of processors to use with multi-processing. The cap is multiprocessing.cpu_count(). Use at your own discretion.
  18. 'useFileNumbering': Use 0.png -> X.png numbering or not. If you want to just use the render farm file names, set this to false.
  19. 'enableAudio': BETA FEATURE. You can now add specific audio files per layer. See Add Specific Audio Trait Section for more info.
  20. 'numLoopMP4': Number of times to loop mp4.
  21. 'generateThumbnail': BETA - Flag to generate thumbnail images. thumbnailHeight and thumbnailWidth are the corresponding flags. ETH JSON supports image and animation_url, image is thumbnail preview image on the feed, animation_url is the image to pull from when the user clicks. thumbnailOutputType is the output type for the thumbnail (you can have gif as thumbnail and mp4 as regular NFT).
  22. 'generatePFP': BETA - Flag to save one individual frame as a PFP using pfpFrameNumber.
  23. 'pfpFrameNumber': BETA - Select which frame to use, starts at 0 (being the first frame).
  24. 'animationUri': BETA - Animation URI used in the toggle HTML features HTML Animation Toggle.

Update step2_spritesheet_to_generative_sheet/src/config.js with your layerConfigurations. If you want the basic configuration, just edit layersOrder, but if you want to take advantage of nftchef's repo, then scroll through the file for some examples and modify layerConfigurations accordingly.

Your output gifs will appear in build/gif, and your output MP4 will appear in build/mp4. The ETH JSON will appear in build/json. Try it yourself with the default settings and layers!

If you want to switch between generating GIFs vs. MP4, you can change the global_config.json and just run make step3.

How does it work?

Step 1

In order to get nftchef's Generative Gif Engine to work, the input layers needs to be in Sprite Sheet. However this is tedious and unintuitive for many artists who use tools that export individual images.

Step 1 simply converts individual images to spritesheets with the rarity percentage. You provide the various layers in the /layers folder with the rarity in the folder name. Each image should be numbered from 0 -> X, and only accepts .png.

If you do not include the rarity weight in the attribute folder name, that attribute will be ignored. These need to be integers. If you want decimal %s, multiple all the rarities by 10 or 100.

You can provide any number of frames in each layer folder, the code will repeat them up until it hits numberOfFrames. It will also trim any that have too many frames.

Example layers folder structure with four layers and two traits each layer:

layers
└───Background
β”‚   └───Grey#50
β”‚       β”‚   0.png
β”‚   └───Pink#50
β”‚       β”‚   0.png
└───Ball
β”‚   └───Blue#50
β”‚       β”‚   0.png
β”‚       β”‚   1.png
β”‚       β”‚   2.png
β”‚       β”‚   ...
β”‚   └───Green#50
β”‚       β”‚   0.png
β”‚       β”‚   1.png
β”‚       β”‚   2.png
β”‚       β”‚   ...
└───Hat
β”‚   └───Birthday#50
β”‚       β”‚   0.png
β”‚       β”‚   1.png
β”‚       β”‚   2.png
β”‚       β”‚   ...
β”‚   └───Cowboy#50
β”‚       β”‚   0.png
β”‚       β”‚   1.png
β”‚       β”‚   2.png
β”‚       β”‚   ...
└───Landscape
β”‚   └───Cupcake#50
β”‚       β”‚   0.png
β”‚   └───Green Tower#50
β”‚       β”‚   0.png

Example layer:

Background:

Grey:

Pink:

Ball:

Blue:

...

Green:

...

Hat:

Birthday:

...

Cowboy:

...

Landscape:

Cupcake:

Green Tower:

I am using python here instead of javascript libraries because I have found that image processing using PIL is much faster and without lossy quality than javascript. These benefits are much clearer in step 3.

You can run only step1 by running:

    make step1

This will convert the pngs into spritesheets and the output will look something like this:

Output:

Background:

Grey#50.png:

Pink#50.png:

Ball:

Blue#50.png:

Green#50.png:

Hat:

Birthday#50.png:

Cowboy#50.png:

Landscape:

Cupcake#50.png:

Green Tower#50.png:

EDIT tool now supports z-index/stacking, grouping and if-then statements. See nftchef's docs for more information. The layers in this step will have to match the format expected in step 2. See the example layer folders for some more info.

EDIT tool now supports gif layers. You can provide layers as gifs and the code will split the gif into frames. See layers_gif_example. It will create a temp folder in step1_layers_to_spritesheet/temp with the resulting separate frames, and then will parse through that folder to create the output. Make sure numberOfFrames is set in global_config.json.

Step 2

Step 2 takes the spritesheets from step 1 and generates all possible combinations based on rarity. This is where all the magic happens! The output is a bunch of spritesheets with all the layers layered on top of each other.

The original idea came from MichaPipo's Generative Gif Engine but now most of the code in this step is forked from nftchef's Generative Engine which is forked from HashLips Generative Art Engine. Please check out Hashlip's πŸ“Ί Youtube / πŸ‘„ Discord / 🐦 Twitter / ℹ️ Website for a more in depth explanation on how the generative process works.

I recently modified this section to use the code from nftchef's Generative Engine which adds the following features:

You will need to update global_config.json and also update layerConfigurations in step2_spritesheet_to_generative_sheet/src/config.js.

You can run only step 2 by running:

    make step2

Example output with the layers folder (only first 4 displayed, but there are 16 total):

Example output with the layers_z_index folder:

Step 3

Step 3 takes the spritesheets from step 2 and creates gifs/MP4. Initially I used PIL, but found some issues with pixel quality.

In MichaPipo's original repo, they used javascript libraries to create the gifs. These copied pixel by pixel, and the logic was a bit complicated. Creating just 15 gifs would take 4 minutes, and I noticed some of the pixel hex colors were off. Also depending on CPU usage, the program would crash. I spent days debugging, when I just decided to start from scratch in another language.

I then tried imageio, and a few Python libraries, but they all had some issues generating gifs.

I spent weeks finding the best tool for this job, and came across gifski. This creates incredibly clean gifs and worked the best.

Now, generating 15 gifs takes < 30 seconds and renders with perfect pixel quality!

You can change the framesPerSecond in global_config.json and you can run only step 3 by running:

    make step3

This allows you to not have to regenerate everything to play around with fps.

Example output with all 16 permutations (click on each gif for the 1000x1000 version):

Some metrics:

MichaPipo's Generative Gif Engine:

New Generative Gif Engine:

NOTE imageio was by far the best Python library, so I added it as an option in case you don't want to download gifski. imageio will work for most pixel art and I know some people had issues with gifski on Linux (not Windows or Mac).

You can set which gif tool to use in global_config.json by setting gifTool to either gifski (default) or imageio.

If you want to switch between generating gif vs. MP4, you need to change outputType to mp4 and only run make step3.

NFTChef improvements: z-index/stacking, grouping, if-then statements, and incompatibilities

Tool now supports z-index/stacking, grouping, if-then statements, and incompatibilities. See nftchef's docs for more information.

TLDR if you don't want to read the doc:

Adding specific audio per trait

πŸ§ͺ BETA FEATURE

You can now add specific audio per trait. For example if you want wind noises with a wind background, and forest noises with a forest background.

Just put the audio file in the corresponding layer folder, and step3 will take that and put it on the mp4. You can see an example in the layers_audio folder. Try it out by setting layersFolder to layers_audio and enableAudio to true, then run make all. The mp4 will be the length of frames and the audio will get truncated if it is too long.

The tool supports mp3, wav, and m4a. If there are multiple audio files for the same NFT, it will combine the audio files and overlap them.

Extend existing collection into GIF/MP4

πŸ§ͺ BETA FEATURE

Video Walkthrough

If you have existing metadata for an existing collection and want to either create a new collection with GIFs/MP4 or send GIF/MP4 version of the static image to holders, this feature is for you! OR if you want to export as a spritesheet that can be imported into a pixel metaverse, this feature is for you!

There are a few configurations to you can use the tool:

  1. If you already have a _dna.json generated by NFT Chef's repo, and a _metadata.json file which contains all the JSON files. Load the _dna.json into the build folder, and load the _metadata.json into the build/json folder. Setup your layers following the format above. Setup global_config.json and config.js and run make regenerate. You can regenerate in parts by following instructions in Generate entire collection in parts except edit regenerate.py instead of all.py. This is the most accurate and consistent way of generating GIFs based on existing layers and will work with NFT Chef's features.
  2. If you generated using Hashlips' art engine, you won't have a _dna.json. You will only have _metadata.json which contains all the JSON files. Load this into the build/json folder, setup layers, setup global_config.json, config.js and run make regenerate. This under the hood attempts to regenerate the DNA based on the JSON. This should work, but there may be features that are not backwards compatible so let me know if you come across such a case.
  3. You don't have a _metadata.json file. Load all the individual .json files into build/json. Setup layers, setup global_config.json, config.js and run make regenerate. This is more annoying to do (if you have a ton of files), but will regenerate the _metadata.json, the _dna.json, and then regenerate the collection.

If you only want to regenerate spritesheets, you can set SKIP_STEP_ONE to True and SKIP_STEP_THREE to True in regenerate.py. Then instead of putting your layers in the layers folder, you put them in step1_layers_to_spritesheet/output as an entire layer, and then run make regenerate. The spritesheets will be in step2_spritesheet_to_generative_sheet/output.

If you need more than 32 frames at 1000x1000, follow the batches configuration and then run make regenerate. This will only work if you are doing all the steps and not skipping any.

Please let me know if you have any issues or use cases I did not think of.

Rarity stats

You can check the rarity stats of your collection with:

    make rarity

Exclude a layer from DNA

If you want to have a layer ignored in the DNA uniqueness check, you can set bypassDNA: true in the options object. This has the effect of making sure the rest of the traits are unique while not considering the Background Layers as traits, for example. The layers are included in the final image.

layersOrder: [
      { name: "Background" },
      { name: "Background" ,
        options: {
          bypassDNA: false;
        }
      },

Provenance Hash Generation

If you need to generate a provenance hash (and, yes, you should, read about it here ),

run the following util

make provenance

This will add a imageHash to each .json file and then concatenate them and hash the file value into one string which is the provenance hash.

The Provenance information is saved to the build directory in _provenance.json. This file contains the final provenance hash as well as the (long) concatenated hash string.

*Note, if you regenerate the gifs, You will also need to regenerate this hash.

Remove trait

If you need to remove a trait from the generated attributes for ALL the generated metadata .json files, you can use the removeTrait util command.

cd step2_spritesheet_to_generative_sheet && node utils/removeTrait.js "Trait Name"

If you would like to print additional logging, use the -d flag

cd step2_spritesheet_to_generative_sheet && node utils/removeTrait.js "Trait Name" -d

Update your metadata info

You can change the description and base Uri of your metadata even after running the code by updating global_config.json and running:

    make update_json

Randomly Insert Rare items - Replace Util

If you would like to manually add 'hand drawn' or unique versions into the pool of generated items, this utility takes a source folder (of your new artwork) and inserts it into the build directory, assigning them to random id's.

Requirements

example:

β”œβ”€β”€ ultraRares
β”‚   β”œβ”€β”€ gifs
β”‚   β”‚   β”œβ”€β”€ 0.gif
β”‚   β”‚   └── 1.gif
β”‚   └── json
β”‚       β”œβ”€β”€ 0.json
β”‚       └── 1.json

You must have matching json files for each of your images.

Setting up the JSON.

Because this script randomizes which tokens to replace/place, it is important to update the metadata properly with the resulting tokenId #.

Everywhere you need the edition number in the metadata should use the ## identifier.

  "edition": "##",

Don't forget the image URI!

  "name": "## super rare sunburn ",
  "image": "ipfs://NewUriToReplace/##.png",
  "edition": "##",

Running

Run with make replace. If you need to replace the folder name, you may have to edit the Makefile directly with the folder.

Note this will not update _dna.json because these new JSONs don't have DNA. This will modify _metadata.json though.

Solana metadata

πŸ§ͺ BETA FEATURE

After running make all you can run generate the Solana metadata in two steps:

Most of the code comes from nftchef.

I have not tried this on any test net or production Solana chain, so please flag any issues or create a PR to fix them!

Tezos metadata

πŸ§ͺ BETA FEATURE

I have not tried this on any test net or production Tezos chain, so please flag any issues or create a PR to fix them!

See Tezos README for more information.

Batching

Do you want higher resolution, more frames, and larger gifs/MP4? Batching is for you! Currently step2 is limited by 32000 pixel files, so in order to get around this we must batch the entire process into chunks and then combine them at the end.

Set useBatches in global_config.json to true and then set numFramesPerBatch to a smaller frame batch. NOTE try testing different frames per batch to see if rendering a smaller number of editions is faster or slower. Some users have said smaller batches renders faster.

This works for odd number of frames as well, ex. 35 total frames, and 12 frames per batch, it will generate 2 batches of 12 frames, and 1 batch of 11 frames automatically.

Then run make all which runs python3 all.py. This under the hood generates the JSON metadata for the first batch and then regenerates the next batches based on the existing JSON.

HTML Toggle Animation

πŸ§ͺ BETA FEATURE

Do you want someway for users to toggle between static and animated images (similar to Little Lemon Friends)?

Check out README_Assets/html/0.html for an example (open it in any browser and click the icon on the top left). Markdown doesn't allow embedding all HTML tags and javascript so you'll have to open it separately.

Steps to generate HTML:

  1. Have all your animated gifs/MP4s in the build/gifs or build/mp4 folder. You can do this by running make all or just copying them into the folders.
  2. Have all your PFPs in the build/pfps folder. You can do this by running make all with generatePFP set to true. You can pick which frame to be your PFPs in pfpFrameNumber. OR you can just drag the PFPs if you already have them generated.
  3. Change the logo to your logo in generate_html/logo.png. It has to be called logo.png all lowercase.
  4. Run make html. All the html files should be in build/html. You can change styling by editing generate_html/template.html.
  5. After uploading the HTML files to IPFS or Pinata, change the animationUri in global_config.json, then run make update_json.

Generate entire collection in parts

πŸ§ͺ BETA FEATURE

Let's say you want to generate a 10k collection with 100 frames each. Most computers don't have enough space to handle hundreds of GB of data, plus the whole process can take days and might get interrupted halfway.

Instead of paying for a remote server and having to pay for tons of storage, now you can run the whole generation process locally in parts! This is different than the "batching" mentioned in previous sections, which batches the frames into smaller batches. Generating in parts means you can generate only part of the collection at a time.

For example, let's say you have a 10k collection with 120 frames. Your global config might look like:

{
        "totalSupply": 10000,
        ...
        "startIndex": 0,
        ...
        "useBatches": true,
        "numFramesPerBatch": 20
}

Now instead of just running make all which will most likely error, you can genereate only 1k editions at a time. You will need to edit all.py.

Look for START_EDITION and END_EDITION. These are going to be which range of editions you want to generate. For example first we could generate 0 - 1000. Edit the file with START_EDITION = 0, and END_EDITION = 1000. This under the hood will generate all 10K JSON files, but only generate the first 1K. You can check rarity and other metadata info now. NOTE END_EDITION is EXCLUSIVE, meaning this will only generate 0 - 999 (total of 1000).

After this finishes, move the build folder to somewhere else on your computer, or an external hard drive, and start generating START_EDITION = 1000 and END_EDITION = 2000. This will generate 1000 - 1999. Repeat the process until (making sure to move the files out of the build folder), START_EDITION = 9000 to END_EDITION = 10000.

Preview Gif/MP4

If you want a preview gif/MP4 of a subset of gifs (like Hashlips), run

make preview

This will output preview.gif/preview.mp4 in the build folder. The default number of previews is 4 but you can change this in step3_generative_sheet_to_output/preview.py at the top NUM_PREVIEW_OUTPUT. Currently it will randomly select the gifs/MP4, if you want to output the first X, set SORT_ORDER to OrderEnum.ASC and if you want to output the last X, set SORT_ORDER to OrderEnum.DESC.

FAQ

Q: Why did you decide to use Python for step 1 and step 3?

A: I found that Python PIL work better and faster than JS libraries, and the code is simpler for me. Initially I tried PIL, imageio, and a few Python libraries, but they all had issues generating gifs. I spent weeks finding the best tool for this job, and came across gifski. This creates incredibly clean gifs and worked the best.

My philosophy is pick the right tool for the right job. If someone finds a better library for this specific job, then let me know!

Q: Why didn't you use Python for step 2?

A: The NFT dev community which writes the complicated logic for generative art mainly codes in javascript. I want to make it easy to update my code and incorporate the best features of other repos as easily as possible, and porting everything to Python would be a pain. You can imagine step 1 and step 3 are just helper tools in Python, and step 2 is where most of the business logic comes from.

Q: What file types do you support?

Input type: gif or png

Output type: gif or MP4

Q: What blockchains do you support?

Ethereum, Solana, Tezos.

Q: I have issues on Windows what should I do?

Check the discussion board, or ask Discord for more issues. Common problems can be found in the installation section.

Need more help?

Be sure to follow me for more updates on this project:

Twitter

GitHub

Medium

My ETH address is 0x4233EfcB109BF6618071759335a7b9ab84F2F4f3 if you feel like being generous :). I just quit my job to work on NFTs full time so anything is appreciated.

If you want to see this code in action, we used it for my fitness and mental health project Fit Friends (now shut down). Check it out here:

Twitter

Website

If you have any questions try asking the Hashlips' Discord. We have a separate channel just for this repo! Go to #animated-art-engine.

Hashlips Discord