Tomes that finish fast fail to respond when another task is blocking IO

spellshift / realm

Realm is a cross platform Red Team engagement platform with a focus on automation and reliability.

GNU General Public License v3.0

419 stars 31 forks source link

Describe the bug When a tome is performing an IO blocking operation like: sleeping, or making a network connection it can prevent other tomes from sending output back to tavern. This only seems to affect tomes that finish quickly (likely within the time between claim_task and report.) "fast" Tomes that have had this issue:

Get env
Shell execute cmd: id
Process list match: *

/realm/implants/imix/src/agent.rs:78

    /*
     * Callback once using the configured client to claim new tasks and report available output.
     */
    pub async fn callback(&mut self) -> Result<()> {
        let transport = GRPC::new(self.cfg.callback_uri.clone(), self.cfg.proxy_uri.clone())?;
        self.claim_tasks(transport.clone()).await?;
        self.report(transport.clone()).await?;

        Ok(())
    }

To Reproduce

Create an IO blocking tome:
```
def main():
time.sleep(999999)
```

main()


2. Import it to tavern - I temporarily replaced `realm/tavern/tomes/example/main.eldritch` with the above and the start tavern.
3. Run a "fast" tome to make sure it works
4. Run the IO blocking tome
<img width="1432" alt="image" src="https://github.com/spellshift/realm/assets/7121375/8b0ee442-f566-4f19-bebc-76b1d83dd14d">

5. Run a "fast" tome again and see how it stays in the `Queued` state
<img width="1452" alt="image" src="https://github.com/spellshift/realm/assets/7121375/90b65653-a790-4b5f-a1be-07815de0469e">

6. Run a slow tome and see that it finishes as expceted
<img width="1428" alt="image" src="https://github.com/spellshift/realm/assets/7121375/af7149da-e606-43a6-b4eb-da0d67baa56e">

**Expected behavior**
Tomes should never interfere with each other. Even if a tome is infinitely blocking other tomes should be able to complete.

Logs

Success case (slow execution)

2024-04-19T00:29:41.178Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:29:46.283Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:29:51.385Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:29:56.490Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:01.492Z INFO imix::agent > 1 tasks claimed 2024-04-19T00:30:01.492Z INFO imix::agent > spawned task execution for id=42949672965 2024-04-19T00:30:01.493Z INFO eldritch::runtime::eval > evaluating tome (task_id=42949672965) 2024-04-19T00:30:01.693Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:01.696Z INFO imix::task > message success (task_id=42949672965,msg=ReportAggOutput) 2024-04-19T00:30:06.502Z INFO eldritch::runtime::eval > tome evaluation successful (task_id=42949672965) 2024-04-19T00:30:06.798Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:06.801Z INFO imix::task > message success (task_id=42949672965,msg=ReportAggOutput) 2024-04-19T00:30:11.901Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:17.006Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:22.109Z INFO imix::agent > callback complete (duration=0s, sleep=5s)

Failure case (fast execution)

2024-04-19T00:30:47.631Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:52.736Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:30:57.839Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:31:02.842Z INFO imix::agent > 1 tasks claimed 2024-04-19T00:31:02.842Z INFO imix::agent > spawned task execution for id=42949672966 2024-04-19T00:31:02.842Z INFO eldritch::runtime::eval > evaluating tome (task_id=42949672966) 2024-04-19T00:31:02.846Z INFO eldritch::runtime::eval > tome evaluation successful (task_id=42949672966) 2024-04-19T00:31:02.942Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:31:08.048Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:31:13.150Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:31:18.254Z INFO imix::agent > callback complete (duration=0s, sleep=5s) 2024-04-19T00:31:23.357Z INFO imix::agent > callback complete (duration=0s, sleep=5s)

spellshift / realm