OOM (Out of Memory) Issues

[marcinguy]

Hi,

experiencing some OOM issues while running in Docker on powerful box with 84 GB Ram.

[431166.569778] Out of memory: Killed process 3400451 (sys) total-vm:1160377588kB, anon-rss:77285872kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:164212kB oom_score_adj:0

Tried to set stack.yml

jobs: 1

Is this setting it to one job? I see all CPU cores are in use ....

Anything else I can do to reduce memory usage, slow down the processing but avoid OOM?

How many threads is it actually starting? As many as cores? Can you reference some code link?

Sorry if this is little bit off topic/general, but there is little info about Haskell and your project, or is it me that is Haskell beginner. Those question can help folks running it.

Thanks,

[marcinguy]

Maybe related: https://github.com/PLSysSec/sys/issues/20

[marcinguy]

Wondering if:

https://github.com/PLSysSec/sys/blob/821c4d7cf924e68838c128cbe824be46c9955416/src/Static/CheckerConfigDef.hs

      cfgBlockBound          :: Int,
                             -- ^ How many blocks should we explore
      cfgLoopBound           :: Int,
                             -- ^ How many times through loops?

Impact RAM usage and what is the balance, balanced value here for each Checker? Are defaults good for big projects?

[marcinguy]

If this helps, there are my LL file sizes and occurences:

https://github.com/marcinguy/public/blob/master/files-occurences-sizes.txt

[marcinguy]

I get results for files under < 1 MB (ll files) on modern laptop (Core i5, 20 GB Ram), with bigger files I seem to get OOMs

[deian]

In our experience the parsing stage is kind of gross and slow. Yes the block and loop bound matter too. I'm not sure there is a bug here. You're asking good questions, but some of these I think you can take a stab at answering them and we'd be happy to incorporate documentation/findings

[marcinguy]

Thanks. Would love to. Pity I never did anything in Haskell. Seems like stack exec sys +RTS -N1 -- -c heapoob -e ll -d /folder helps a little bit here, untill OOM happens later on P.S Total Haskell beginner

[marcinguy]

@deian Can you give a hint how to run in on several cores only? Limited resources? So that it does not OOM that quickly?

[mlfbrown]

Aha! This is related to #4 --- @deian is right that we do the dumbest thing right now. If you bring down the block bound it should help, I'm guessing the loop bound should already be low. I can fix this in about three weeks (sorry! have been meaning to for a while!), otherwise what will help is (1) reducing the number of paths that static flags (2) reducing the block bound.

[marcinguy]

Thanks @mlfbrown @deian

Reduced the block bound to 1, not sure how to do (1) paths that static flags and where is it defined, but it still OOMs.

The destination machine I work on have many cores/CPUs, trying to cap it to several cores (docker run --cpuset-cpus="0-N"), than I hope to have more RAM per cores used. Not sure if this will help, how the threading works in Haskell and Sys. Will update later.

Does Sys, Haskell starts one thread per core based on amount of cores? Memory should be shared between threads, but I guess with more cores, you will need more memory??? Speed will be faster with more cores, but you need more Memory. So I want to sacrifice speed for completeness (no OOMs). Is this assumption correct?

[marcinguy]

Capping CPUs/cores in Docker seem to work, but Sys starts so many Threads as cores .... will see how this will work. Will update.

[marcinguy]

Trying this ... will see if this helps.

import Control.Concurrent(setNumCapabilities)

fork :: Checker a b () -> Checker a b ()
fork act = do
  s0 <- get
  void $ liftIO $ do
    setNumCapabilities 1
    forkIO (curTGroup s0) $ evalChecker act s0

Can make a PR to add this as an option to set number of threads vs getting cores amount.

Not sure how to get it from command line parameter and pass it on.

Here is my PR:

https://github.com/PLSysSec/sys/pull/22

[marcinguy]

Nah ... running it now with Bash to skip files when it OOMs and continue where it left off.

On my side the issue is closed. Feel free to close if you wish.

Thanks for your help. Looking forward to future improvements.

#!/bin/bash
export LOG_LEVEL=debug

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c uninit -e ll -d "$REPLY" >> /local/uninit_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c heapoob -e ll -d "$REPLY" >> /local/heapoob_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c concroob -e ll -d "$REPLY" >> /local/concroob_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c userinput -e ll -d "$REPLY" >> /local/userinput_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )

find /local/ -depth -name "*.done" -exec sh -c 'mv "$1" "${1%.ll.done}.ll"' _ {} \;

while IFS= read -r -d '' -u 9
do
        stack exec sys -- -c uaf  -e ll -d "$REPLY" >> /local/uaf_results.txt
        mv "$REPLY" "$REPLY".done
done 9< <( find /local/ -type f -name "*.ll" -exec printf '%s\0' {} + )