[Feature request]: Programmatic use and API

[ebceu4]

I would like to request a feature that allows for the saving and loading of pipelines as JSON. This will enable users to create complex and advanced pipelines using the graph/nodes/flowchart based interface and then leverage the visually built pipelines programmatically or via API through a runner

[comfyanonymous]

See: https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

[IcelandicIcecream]

Is there a guide on how the API works

[lunaticer]

I also want to use workflow struct my special API,because I can't process batch images now.It will make the workflow more powerful.

[icelandno1]

That's a good question, and I'd like to implement this functionality by calling the ComfyUI workflow directly from an external application, which eliminates the hassle of using the stable diffusion api.

[Razorbob]

I would also love to have an API. If possible also an OpenApi/Swagger definition would be awesome, so you could generate a client in different languages.

[misza222]

+1

[comfyanonymous]

if you update to the latest you can "enable dev mode options" in the settings (gear beside the "Queue Size: ") this will enable a button on the UI to save workflows in api format.

You can then use these api jsons in these script examples: https://github.com/comfyanonymous/ComfyUI/tree/master/script_examples

[tomermesser]

Hi, Thanks for adding the "API Format" option it is super helpful !

[fidecastro]

I took a look at how the websocket service works, and created a simple class to use the already existing API. It facilitated my workflow a lot and maybe some of you can find it useful.

https://github.com/fidecastro/ComfyUI-serverless/

[p0mad]

I'm Stucking in here! https://github.com/comfyanonymous/ComfyUI/issues/1083#issuecomment-1699082974

Can you give me a tought?

[kiriri]

/I could not find an api documentation, so here's what I gleamed from the source code and everyone's comments :

GET

/history Last queued renders, including all nodes used in the prompt, and all output images.

/history/{UID} A specific history entry by uid. When calling prompt, the result will contain this uid.

/view?filename={FILENAME} Access the images from history via this.

/extensions paths to the javascript files for each extension.

/embeddings Presumably a list of all embeddings. These aren't available as inputs in any Node, unlike models (see OTHERS).

/object_info List of all types of nodes that are available. (Not the actual node instances, just the node types.)
This exposes a lot of important enumerable values in the form of input types.

/queue What's happening or queued at the moment. Surprisingly comprehensive.

/system_stats os, python version, torch devices (like gpus) and vram used.

POST

/prompt https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

The json data payload must be stored under the name "prompt". In JS, it would look like so :

const res = await axios.post('http://127.0.0.1:8188/prompt',{
    prompt:json_data_object 
});

You can get an example of the json_data_object by enabling Dev Mode in the ComfyUI settings, and then clicking the newly added export button.

/interrupt Interrupts the execution of the running prompt and starts the next one in the queue.

/queue /history In POST, these routes should accept the payloads clear:true and delete:number[] .

/upload/image /upload/mask This uploads an image and returns a short path under which the image can be referred to in comfyui. Takes payload 'image' in body.

SYNTAX

The following typescript definitions are incomplete and may be wrong entirely.
Please leave a comment if you spot errors or if you have improvements.

// This is what apis like "prompt" expect.
export type JSON_ComfyGraph = {
    [uid: string|number]: JSON_ComfyNode
}

// A node instance.
export type JSON_ComfyNode = {
    // The type of the node as found in 'GET /object_info' by key
    class_type: string;
    // All incoming connections
    inputs: {
        [input_name: string]: JSON_ValueRef;
    }
}

// Any value that can is passed into a ComfyNode
// This can be either an atomic value, like a string or a number, or a reference to another node's output.
export type JSON_ValueRef = [target_node_uid: string, output_socket_index: number] | string | number | boolean

// All possible input/output types. This type is incomplete.
// string[] represents an enum dropdown.
export type JSON_ComfyPrimitive = "MODEL" | "CLIP" | "STYLE_MODEL" | "CLIP_VISION_OUTPUT" | "CONTROL_NET" | "UPSCALE_MODEL" | "CLIP_VISION" | "VAE" | "LATENT" | "INT" | "FLOAT" | "IMAGE" | "TEXT" | "CONDITIONING" | "EXTRA_PNGINFO" | "MASK" | string[];

// Some types accept options like restrictions or special ui operations. So they are wrapped in an array of type + options
export type JSON_ComfyPrimitiveDeclaration = 
// An empty array means the name of the output/input is the same as the type. Case insensitive. Eg an output named 'latent' defaults to type "LATENT".
[] 
| [JSON_ComfyPrimitive]
| [
    "INT" | "FLOAT",
    {
        default?: number,
        min?: number,
        max?: number,
        step?: number,
        round?: number
    }
]
| [
    "TEXT"|string[],
    {
        multiline?:boolean,
        default?:string
    }
]
| [
    "IMAGE"|string[],
    {
        image_upload?:boolean,
        default?:string
    }
]

// This is what apis like "object_info" return.
export type JSON_ComfyNodeTypes = {
    [key: string]: {
        // This is what creates the submenus in the context menu.
        category: string,
        // Verbal description of the node for use in the ui.
        description: string,
        // The name which should be displayed in the editor.
        display_name: string,
        input: {
            // List of all input types
            required: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            },
            hidden: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            }
        },
        // Unique name
        name: string,
        // Unique names of all the outputs
        output: string[],
        // Is the output a list?
        output_is_list: boolean[],
        // Labels of all the outputs
        output_name: string[],
        // Is this node an output? Useful for evaluating if a graph is valid.
        output_node: boolean;
    }
}

// This is what apis like "history" return.
export type JSON_History = {
    [uid: string]: JSON_HistoryEntry;
}

export type JSON_HistoryEntry = {
    prompt: [id_maybe: number, uid:string, list_of_used_nodes:JSON_ComfyGraph, no_idea:any, no_idea2:[string]];
    outputs: {
        images: JSON_ImageReference[];
    }[];
}

export type JSON_ImageReference = {
    filename: string;
    subfolder: string;
    type: "output";
}

OTHER

You can retrieve the list of all available models from the CheckpointLoader object. GET /object_info Then CheckpointLoader->input->required->ckpt_name->0 .

[darkcurrent]

I took a look at how the websocket service works, and created a simple class to use the already existing API. It facilitated my workflow a lot and maybe some of you can find it useful.

https://github.com/fidecastro/ComfyUI-serverless/

It's a great work

[zephirusgit]

Could anyone give me any functional example of the use of the API? What is complete and that has an example embedded as those of the Save in Api Format? I can't run the Basic Example:/

[zephirusgit]

si actualizas a lo último puedes "solar las opciones de modo dev" en la configuración (envejecer junto al "Queue Size: ") esto permitirá un botón en la interfaz de usuario para guardar flujos de trabajo en formato api.

A continuación puede utilizar estos api jsons en estos ejemplos de script: https://github.com/comfyanonymous/ComfyU/tree/master/script-examples

Could anyone give me any functional example of the use of the API? What is complete and that has an example embedded as those of the Save in Api Format? I can't run the Basic Example:/

[rsandagon]

For sample usage you may check this small project I've created. https://github.com/rsandagon/comfyui-batch-image-generation

For websocket handling, check here

For API endpoint, check here

You need to have the same client_id for both API and websocket to get proper websocket updates on the workflow.

[rvion]

you can also look at https://github.com/rvion/CushyStudio

It is probably the biggest custom frontend around.

I maintain the most comprehensive set of typings possible, along a higher level type-safe typescript ComfyUI SDK to build workflows with all possible goodies you can imagine.

CushyStudio also generates a whole type-safe SDK for your specific ComfyUI setup, with every custom node, and even model you have installed

• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L6789
• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L5324
• https://gist.github.com/rvion/f68635427c7d5e9b0846121e4d9e87a7#file-global-d-ts-L5190

writing workflows and running them is super smooth.

Cushy also feature a scripting engine with hot reload, and a custom gradio-like framework (simpler and type-safe) to build custom apps with UI on top of the ComfyUI SDK.

Only reason it's not popular yet is that it's not released yet (so no one knows about it), and that it probably has a few quirks here and there. but maybe you'll have fun trying it out. feel free to reach out on discord if you need help:)

[rvion]

Cushy also includes higher level API / typings for comfy manager, and host management too, (and other non-comfy things that works well with ComfyUI, like a full programmatic image building API to build masks, etc)

[zephirusgit]

Finally I found a page, which explained how to use it in Python and it worked for me https://medium.com/@yushantripleseven/comfyui-websockets-api-part-1-618175802d5a

https://medium.com/@yushantripleseven/comfyui-using-the-api-part-2-daac17fd2727

[hippogamesunity]

/I could not find an api documentation, so here's what I gleamed from the source code and everyone's comments :

GET

/history Last queued renders, including all nodes used in the prompt, and all output images.

/history/{UID} A specific history entry by uid. When calling prompt, the result will contain this uid.

/view?filename={FILENAME} Access the images from history via this.

/extensions paths to the javascript files for each extension.

/embeddings Presumably a list of all embeddings. These aren't available as inputs in any Node, unlike models (see OTHERS).

/object_info List of all types of nodes that are available. (Not the actual node instances, just the node types.) This exposes a lot of important enumerable values in the form of input types.

/queue What's happening or queued at the moment. Surprisingly comprehensive.

/system_stats os, python version, torch devices (like gpus) and vram used.

POST

/prompt https://github.com/comfyanonymous/ComfyUI/blob/master/script_examples/basic_api_example.py

The json data payload must be stored under the name "prompt". In JS, it would look like so :

const res = await axios.post('http://127.0.0.1:8188/prompt',{
  prompt:json_data_object 
});

You can get an example of the json_data_object by enabling Dev Mode in the ComfyUI settings, and then clicking the newly added export button.

/interrupt Interrupts the execution of the running prompt and starts the next one in the queue.

/queue /history In POST, these routes should accept the payloads clear:true and delete:number[] .

/upload/image /upload/mask This uploads an image and returns a short path under which the image can be referred to in comfyui. Takes payload 'image' in body.

SYNTAX

The following typescript definitions are incomplete and may be wrong entirely. Please leave a comment if you spot errors or if you have improvements.

// This is what apis like "prompt" expect.
export type JSON_ComfyGraph = {
    [uid: string|number]: JSON_ComfyNode
}

// A node instance.
export type JSON_ComfyNode = {
    // The type of the node as found in 'GET /object_info' by key
    class_type: string;
    // All incoming connections
    inputs: {
        [input_name: string]: JSON_ValueRef;
    }
}

// Any value that can is passed into a ComfyNode
// This can be either an atomic value, like a string or a number, or a reference to another node's output.
export type JSON_ValueRef = [target_node_uid: string, output_socket_index: number] | string | number | boolean

// All possible input/output types. This type is incomplete.
// string[] represents an enum dropdown.
export type JSON_ComfyPrimitive = "MODEL" | "CLIP" | "STYLE_MODEL" | "CLIP_VISION_OUTPUT" | "CONTROL_NET" | "UPSCALE_MODEL" | "CLIP_VISION" | "VAE" | "LATENT" | "INT" | "FLOAT" | "IMAGE" | "TEXT" | "CONDITIONING" | "EXTRA_PNGINFO" | "MASK" | string[];

// Some types accept options like restrictions or special ui operations. So they are wrapped in an array of type + options
export type JSON_ComfyPrimitiveDeclaration = 
// An empty array means the name of the output/input is the same as the type. Case insensitive. Eg an output named 'latent' defaults to type "LATENT".
[] 
| [JSON_ComfyPrimitive]
| [
    "INT" | "FLOAT",
    {
        default?: number,
        min?: number,
        max?: number,
        step?: number,
        round?: number
    }
]
| [
    "TEXT"|string[],
    {
        multiline?:boolean,
        default?:string
    }
]
| [
    "IMAGE"|string[],
    {
        image_upload?:boolean,
        default?:string
    }
]

// This is what apis like "object_info" return.
export type JSON_ComfyNodeTypes = {
    [key: string]: {
        // This is what creates the submenus in the context menu.
        category: string,
        // Verbal description of the node for use in the ui.
        description: string,
        // The name which should be displayed in the editor.
        display_name: string,
        input: {
            // List of all input types
            required: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            },
            hidden: {
                [name: string]: JSON_ComfyPrimitiveDeclaration
            }
        },
        // Unique name
        name: string,
        // Unique names of all the outputs
        output: string[],
        // Is the output a list?
        output_is_list: boolean[],
        // Labels of all the outputs
        output_name: string[],
        // Is this node an output? Useful for evaluating if a graph is valid.
        output_node: boolean;
    }
}

// This is what apis like "history" return.
export type JSON_History = {
    [uid: string]: JSON_HistoryEntry;
}

export type JSON_HistoryEntry = {
    prompt: [id_maybe: number, uid:string, list_of_used_nodes:JSON_ComfyGraph, no_idea:any, no_idea2:[string]];
    outputs: {
        images: JSON_ImageReference[];
    }[];
}

export type JSON_ImageReference = {
    filename: string;
    subfolder: string;
    type: "output";
}

OTHER

You can retrieve the list of all available models from the CheckpointLoader object. GET /object_info Then CheckpointLoader->input->required->ckpt_name->0 .

Thanks, this should be added to wiki)

[hippogamesunity]

Is there any way to clear the output folder with API?

[CrossPr0duct]

@comfyanonymous is there a way to turn off nodes other than setting strength it doesn't seem like the api version of a workflow allows mode modification

[hippogamesunity]

@comfyanonymous is there a way to turn off nodes other than setting strength it doesn't seem like the api version of a workflow allows mode modification

You can create xml workflows on fly and pass them to /promp with POST. Also I've found a 3rd article with API description based on code analysis.

[robinjhuang]

Closing after a week of inactivity.

[teodor-krastev]

Hi there,

is there a way to read via API the front UI parameters like width, height, cfg, etc. before doing anything?

Cheers Theo

[ltdrdata]

Hi there,

is there a way to read via API the front UI parameters like width, height, cfg, etc. before doing anything?

Cheers Theo

The question is incorrect. You need to create the front data (i.e., prompt) and send it via API. The backend doesn't have the front end's data.

[teodor-krastev]

Thank you ltdrdata for your prompt reply. I don't think my question is incorrect, simply the answer is NO. The backend acts as a server to the front end and towards API as clients. Have a lovely weekend! Theo

[teodor-krastev]

For anyone interested, I have found a "dirty" way to read the user web interface. If you execute an image generation from web UI and then get "http://127.0.0.1:8188/history" (without promptId) you will have the parameters from web UI in json format.