Flow Run POST via REST API not getting assoicated with a work queue

[schodge]

First check

• [X] I added a descriptive title to this issue.
• [X] I used the GitHub search to find a similar issue and didn't find it.
• [X] I searched the Prefect documentation for this issue.
• [X] I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

I talked with two Prefect solutions engineers just now, and they suggested this could be a REST API bug. Looking at the backlog, it may be the same as https://github.com/PrefectHQ/prefect/issues/6619

We're using the REST API to verify the pipelines are working correctly in Prefect 2 alongside Prefect 1 as part of our migration since we can't use both clients simultaneously; this issue is blocking our migration.

We've set up a deployment. The deployment has a work queue attached. We've verified we can execute the flow from the UI and it works.

We send a REST flow run POST.

    data = {
        'parameters': params, 
        'flow_id': 'cfd9e6db...',
        'deployment_id': '24583f30...4',
        'infrastructure_document_id': '0181dc52...',
        # tried without the next key entirely, and in "PENDING"
        "state": {"type": "RUNNING"}
    }
    r = requests.post(PREFECT2_URL, json=data, headers={'Authorization': f'Bearer {api_key}'})

(identical behavior via Postman as Python)

We get back a result (see the error box)

There's no work queue populated. The task shows up in the UI without a work queue. Compare below, with the top run submitted via REST and having no work queue and the run below it submitted via the API and having a work queue.

There are no logs whatsoever for the REST runs. There are normal logs for the UI triggered runs.

Reproduction

see above

Error

{
    "id": "585e9cad-...",
    "created": "2023-01-12T21:03:29.623120+00:00",
    "updated": "2023-01-12T21:03:29.625422+00:00",
    "name": "modest-ladybug",
    "flow_id": "cfd9e6db-...",
    "state_id": "1e0299c4-...",
    "deployment_id": "24583f30-...",
    "work_queue_name": None,
    "flow_version": None,
    "parameters": {"documentset_id": "08eff4a9-..."},
    "idempotency_key": None,
    "context": {},
    "empirical_policy": {"max_retries": 0, "retry_delay_seconds": 0.0, "retries": None,
                         "retry_delay": None, "pause_keys": [], "resuming": False},
    "tags": [],
    "parent_task_run_id": None,
    "state_type": "RUNNING",
    "state_name": "Running",
    "run_count": 1,
    "expected_start_time": "2023-01-12T21:03:29.622674+00:00",
    "next_scheduled_start_time": None,
    "start_time": "2023-01-12T21:03:29.622674+00:00",
    "end_time": None,
    "total_run_time": 0.0,
    "estimated_run_time": 0.053743,
    "estimated_start_time_delta": 0.0,
    "auto_scheduled": False,
    "infrastructure_document_id": "0181dc52-...",
    "infrastructure_pid": None,
    "created_by": {"id": "48387560-...", "type": "USER", "display_value": "shaynetomecom"},
    "work_pool_name": None,
    "work_pool_queue_name": None,
    "state": {"id": "1e0299c4-545e-4939-b75e-7b1a88fdf182", "type": "RUNNING", "name": "Running", "timestamp": "2023-01-12T21:03:29.622674+00:00",
            "message": None,    "data": None, "state_details": {
                    "flow_run_id": "585e9cad-...", "task_run_id": None, "child_flow_run_id": None,
                    "scheduled_time": None, "cache_key": None, "cache_expiration": None, "untrackable_result": False, "pause_timeout": None,
                    "pause_reschedule": False, "pause_key": None}
            }
}

### Versions

```Text
This isn't being done via the prefect client, and I don't have access to the containers the agent runs on.

Additional context

No response

[peytonrunyan]

Thanks for the writeup and checking into the existing issues! We'll try to get this squared away shortly.

[peytonrunyan]

@schodge Can you confirm which endpoint you're using?

[schodge]

From the API reference:

https://api.prefect.cloud/api/accounts/45545742.../workspaces/695fb847.../flow_runs/"

In some of the help material, there's a different URL, but I didn't find that to work:

https://discourse.prefect.io/t/how-to-connect-to-prefect-cloud-2-0-restapi/1627/2

https://api.prefect.cloud/api/accounts/45545742.../deployments/24583f30.../create_flow_run

(truncating all the uuids)

[peytonrunyan]

Was it this API reference? https://app.prefect.cloud/api/docs . We may need to update our links somewhere if we have conflicting reference docs.

Here's the format that you should be using: https://api.prefect.cloud/api/accounts/{account_id}/workspaces/{workspace_id}/deployments/{id}/create_flow_run

Here's more information on the body that this endpoint expects: https://app.prefect.cloud/api/docs#tag/Deployments/operation/create_flow_run_from_deployment_api_accounts__account_id__workspaces__workspace_id__deployments__id__create_flow_run_post

[schodge]

It appears the problem is there is what you just mentioned - there is a create flow run from deployment at an endpoint of create_flow_run, and a create flow run and flow_runs

[zanieb]

@schodge it'd be really helpful if you constructed an executable minimal reproducible example that demonstrates your problem. If we cannot copy and paste it to run it, we cannot attempt to reproduce and debug your problem.

I added coverage for this in a test and it succeeds https://github.com/PrefectHQ/prefect/pull/8203

[peytonrunyan]

@schodge I see where that's causing the confusion. Next time the minimum reproducible example will definitely help us diagnose a bit faster. I'll check with the team about seeing if there might be a way to clarify that in our docs.

Does it resolve the issue when you use the deployments endpoint?

[schodge]

Yes, changing the endpoint fixed it. Thanks for the help!

[jackdesert]

How about removing the POST /api/flow_runs endpoint altogether? That would make it clear that we are to use the POST /api/deployments/DEPLOYMENT_ID/create_flow_run endpoint.