Agent is not able to start handlers after killing them

Cyb3r-Monk commented 1 year ago

When the working hours passed, agent kills the handlers. However, when it's time to start the handlers again, agent is not able to start them. When I restart the agent, it starts working again. Some error logs are below. Only Excel doesn't seem to have an issue.

2023/03/16 08:59:51.253|Ghosts.Client.Handlers.Outlook.SendEmailViaOutlook|Creating outlook mail item
2023/03/16 08:59:51.253|Ghosts.Client.Handlers.Outlook.SendEmailViaOutlook|System.Runtime.InteropServices.COMException: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
2023/03/16 08:59:51.253|Ghosts.Client.Handlers.Outlook.SendEmailViaOutlook|Returning - wasSuccessful:False
2023/03/16 08:59:51.253|Ghosts.Client.Handlers.Outlook.ExecuteEvents|DelayAfter sleeping for 3900000 ms
2023/03/16 08:59:54.455|Ghosts.Client.Handlers.BaseBrowserHandler.ExecuteEvents|OpenQA.Selenium.WebDriverException: Could not connect to the remote WebDriver server for URL http://localhost:55719/session/7b63a11284141a24c456419f8d48646d/window.
2023/03/16 08:59:54.456|Ghosts.Client.Infrastructure.WorkingHours.Is|For BrowserChrome: Local time: 08:59:54.4566604 UTC: 08:59:54.4566604 On: 08:09:00 Off: 22:30:00 Overnight? False
2023/03/16 08:59:54.828|Quartz.Logging.LogProviders.NLogLogProvider+NLogLogger.Log|Batch acquisition of 0 triggers

sei-dupdyke commented 1 year ago

Hmm, I can't recreate this, I'll have to look into it further. I'm really starting to hate Outlook across a spectrum of concerns.

Cyb3r-Monk commented 1 year ago

Chrome has the same issue :/ v6 still works without any issues, btw. Shall I try the new Outlook handler in v7? Still using the old handler right now.

Cyb3r-Monk commented 1 year ago

Did some research on the issue and saw some people saying that something had changed after Chrome v98. Some potential solutions there: https://stackoverflow.com/questions/71928501/selenium-webdriver-exception-the-http-request-to-the-remote-webdriver-server-for

Couldn't find anything useful for Outlook. I use Micorosft 365. Maybe it has something to do with the session.Logon().

sei-dupdyke commented 1 year ago

Still can't repro this. Are you saying you are using v6 client?

I can't handle supporting multiple versions, I would ask you to update to v7 if you are having issues in any previous version please.

Cyb3r-Monk commented 1 year ago

It happens in v7. It might be related to the this issue: https://github.com/cmu-sei/GHOSTS/commit/a8146ab084bdd2bfd3a2e4fd1fc8edd4decdee54 Was the fix released? I can update and test again.

Cyb3r-Monk commented 1 year ago

I monitored the situation closely today:

Outlookv2 handler didn't cause any issues this time.
Chrome Browser didn't start(the same WebDriver connection error). I checked the task manager to see if there is any leftover process. GoogleCrashHandler.exe and GoogleCrashHandler64.exe was running. Maybe the agent is not terminating the browser threads properly?

sei-dupdyke commented 1 year ago

What version of chrome? What version of chromedriver?

Seems to be working fine for me - but it might be configuration differences. Are you running headless? incognito?

Cyb3r-Monk commented 1 year ago

Chrome: 111.0.5563.147 Driver: 111.0.5563.64 (latest available)

This is the timeline config, no headless or incognito:

    {
      "HandlerType": "BrowserChrome",
      "Initial": "about:blank",
      "UtcTimeOn": "08:09:00",
      "UtcTimeOff": "20:30:00",
      "HandlerArgs": {
        "isheadless": "false",
        "blockimages": "false",
        "blockstyles": "false",
        "blockflash": "false",
        "blockscripts": "false",
        "stickiness": "65",
        "stickiness-depth-min": "3",
        "stickiness-depth-max": "15",
        "incognito": "false",
        "user-data-dir": "%LOCALAPPDATA%\\Google\\Chrome\\User Data\\",
        "javascript-enable": "true",
        "visited-remember": "5",
    "actions-before-restart": 100,
        "command-line-args": [ "--ignore-certificate-errors" ],
    "delay-jitter": 40,
        "url-replace": [
          {
            "verb": [
              "order",
              "enable",
              "engage"
            ]
          },
          {
            "group": [
              "operations",
              "logistics",
              "medical"
            ]
          },
          {
            "org": [
              "army",
              "command",
              "brigade",
              "battalion"
            ]
          },
          {
            "type": [
              "document",
              "doc",
              "files",
              "vault",
              "filevault"
            ]
          }
        ]
      },
      "Loop": true,
      "TimeLineEvents": [
        {
          "TrackableId": null,
          "Command": "random",
          "CommandArgs": [
            "http://craigslist.org/{org}/{group}/{uuid}/{verb}/{type}/{n}?{c}={now}",
            "https://www.linkedin.com",
            "https://twitter.com",
            "https://www.yahoo.com",
            "https://www.msn.com",
            "https://www.weather.com",
            "https://www.bbc.com",
            "https://www.pinterest.com",
            "https://www.youtube.com",
            "http://wikipedia.org",
            "http://ebay.com",
            "http://craigslist.org",
            "http://medium.com",
            "http://instagram.com",
            "http://imdb.com",
            "http://nytimes.com",
            "http://buzzfeed.com",
            "http://homedepot.com",
            "http://target.com",
            "http://bestbuy.com",
            "http://intuit.com",
            "http://fedex.com",
            "http://ancestry.com",
            "http://swagbucks.com",
            "http://wikimedia.org",
            "http://dailymotion.com",
            "http://nbcsports.com",
            "http://walgreens.com",
            "http://photobucket.com",
            "http://bhphotovideo.com",
            "https://www.bbc.com/sport",
            "http://archive.org",
            "http://merriam-webster.com",
            "http://booking.com",
            "http://bodybuilding.com",
            "http://evite.com",
            "http://careerbuilder.com",
            "http://shareasale.com",
            "http://www.careerjournal.com",
            "http://factcheck.org",
            "http://audacityteam.org",
            "http://amazon.com",
            "http://cnn.com",
            "http://espn.go.com",
            "http://apple.com",
            "http://www.thelayoff.com",
            "http://groupon.com",
            "http://slickdeals.net",
            "http://dailymail.co.uk",
            "http://macys.com",
            "http://deviantart.com",
            "http://theguardian.com",
            "http://goodreads.com",
            "http://okcupid.com",
            "http://bloomberg.com",
            "http://accuweather.com",
            "http://meetup.com",
            "http://mashable.com",
            "http://allrecipes.com",
            "http://rei.com",
            "http://cars.com",
            "http://myway.com",
            "http://stumbleupon.com",
            "http://lifehacker.com",
            "http://ticketmaster.com",
            "http://nba.com",
            "http://consumerreports.org",
            "http://directv.com",
            "http://edmunds.com",
            "http://gamespot.com",
            "http://seekingalpha.com",
            "http://androidcentral.com",
            "http://spotify.com",
            "http://ups.com",
            "http://wordpress.org",
            "http://addthis.com",
            "http://kotaku.com",
            "http://slideshare.net",
            "http://popsugar.com",
            "http://reuters.com",
            "http://fool.com",
            "http://tvguide.com",
            "http://macrumors.com",
            "http://kmart.com",
            "http://speedtest.net",
            "http://stanford.edu",
            "http://trello.com",
            "http://google.com",
            "http://www.yhd.com",
            "http://www.worldairlineawards.com",
            "http://www.womenshealthmag.com",
            "http://www.wolframalpha.com",
            "http://www.vikings.com",
            "http://www.usxpress.com",
            "http://www.usairways.com",
            "http://www.twilert.com",
            "http://www.tvguide.com",
            "http://www.traveltune.com",
            "http://www.travelpod.com",
            "http://www.trackurstatus.com",
            "http://www.toptensocialmedia.com",
            "http://www.toptenreviews.com",
            "http://www.spotify.com",
            "http://www.nike.com",
            "https://www.popsugar.com/moms/",
            "https://www.parents.com/about-us/",
            "https://www.babble.com/",
            "https://www.workingmother.com/",
            "http://www.makeup.com",
            "http://www.lyricsworld.com",
            {
              "Uri": "http://httpbin.org/post",
              "Category": "cat1",
              "Method": "POST",
              "Headers": {
                "1": "a",
                "2": "b"
              },
              "FormValues": {
                "1": "a",
                "2": "b"
              }
            },
            {
              "Uri": "http://httpbin.org/put",
              "Category": "cat1",
              "Method": "PUT",
              "Headers": {
                "1": "a",
                "2": "b"
              },
              "Body": "body"
            },
            {
              "Uri": "http://httpbin.org/delete",
              "Category": "cat1",
              "Method": "DELETE"
            }
          ],
          "DelayAfter": 150000,
          "DelayBefore": 1200000
        }
      ],
      "ScheduleType": "Other",
      "Schedule": null
    }

Cyb3r-Monk commented 1 year ago

I think I've found the root cause for the browser, but I don't know how to fix it. When the browser handler starts, it reads all the URLs in the arguments and stores them. Then it starts the browsing activity. When the browser handler enters into a non-working hour, the chrome driver gets killed but the handler still tries to browse the URLs since it has already read all the URLs and tries to finish the browsing activity. Its' because the handler checks the working hours only when it starts. So, the browser needs to check if it's still in the working hours before browsing any URL from the list.

Cyb3r-Monk commented 1 year ago

Update: Chrome handler code is not the same as Firefox, maybe it has something to do with this issue.

sei-dupdyke commented 1 year ago

Sorry, been all around the world and have not had a chance to look at this. We use FF, and this is what happens, sorry. I'll look soon.

Cyb3r-Monk commented 1 year ago

No worries. I adopted the FF code and disabled killing threads in the workinghours.cs with the help of ChatGPT locally and built it 🤓

sei-dupdyke commented 1 year ago

Chrome is now as close to FF as possible in v7.0.145.

cmu-sei / GHOSTS

Agent is not able to start handlers after killing them #192