search

Picocrypt / CLI

A command-line interface for Picocrypt.
GNU General Public License v3.0
29 stars 2 forks source link

Picocrypt killed during encryption and decryption on OpenBSD 7.6 #21

Open 776d opened 19 hours ago

776d commented 19 hours ago

Picocrypt is killed during encryption and decryption on OpenBSD 7.6.

System Info

% uname -a
OpenBSD <hostname> 7.6 GENERIC.MP#338 amd64
% sysctl hw
...
hw.model=Intel(R) Core(TM) i5-8500T CPU @ 2.10GHz
hw.ncpu=1
hw.physmem=961306624
hw.vendor=QEMU
hw.product=Standard PC (Q35 + ICH9, 2009)
hw.version=pc-q35-8.2
...

Picocrypt installation

% doas pkg_add go
quirks-7.50 signed on 2024-11-20T23:31:32Z
go-1.23.1: ok
% go install github.com/Picocrypt/CLI/picocrypt@latest
go: downloading github.com/Picocrypt/CLI v0.0.0-20241112002509-12591ffeed33
go: downloading github.com/Picocrypt/CLI/picocrypt v0.0.0-20241112002509-12591ffeed33
go: downloading github.com/Picocrypt/infectious v0.0.0-20240830233326-3a050f65f9ec
go: downloading github.com/Picocrypt/serpent v0.0.0-20240830233833-9ad6ab254fd7
go: downloading github.com/schollz/progressbar/v3 v3.17.1
go: downloading golang.org/x/crypto v0.29.0
go: downloading golang.org/x/term v0.26.0
go: downloading golang.org/x/sys v0.27.0
go: downloading github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db
go: downloading github.com/rivo/uniseg v0.4.7
% picocrypt
Usage: picocrypt <item1> [<item2> ...]
Items: can be files, folders, or globs
Flags:
  -f    (decryption) attempt to fix corruption
  -k    (decryption) keep output even if corrupted
  -p    (encryption) use paranoid mode
  -r    (encryption) encode with Reed-Solomon

Encryption

% cat note
astonishing self detective win nerve mature rage block dip draw
panel barrier consumer affair mercy consolidate tropical routine
neel alarm translate reform theorist frown net stroll doctor
aquarium vision chimney solid curl feel joystick harvest lily
white dinner coin huge immune flat straw detail variant run
mixture room implication minimum
% picocrypt note
Password: ******* | Confirm: *******
zsh: killed     picocrypt note
% sysctl vm.loadavg #this command ran in another tmux pane while encryption was running
vm.loadavg=2.25 0.90 0.53
% hexdump -C note.pcv
00000000  76 31 2e 33 34 dd aa c3  c1 3b 42 03 29 c4 ee 30  |v1.34....;B.)..0|
00000010  30 30 30 30 30 30 30 30  30 30 30 30 30 30 00 00  |00000000000000..|
00000020  00 00 00 00 00 00 00 00  00 00 00 00 00 dd f5 18  |................|
00000030  c8 bf 03 55 85 c2 cf 00  1c ba 2a de 66 12 e0 6e  |...U......*.f..n|
00000040  75 19 40 b9 3d c4 a1 c4  bf 83 94 c9 b4 69 ce 7e  |u.@.=........i.~|
00000050  ea 5a 48 c7 f9 d4 ba 29  3e e1 22 f3 c5 0a 73 f5  |.ZH....)>."...s.|
00000060  18 41 0a c5 f5 9b 06 6c  c7 12 fa da cc 0a 01 f4  |.A.....l........|
00000070  4a 29 72 f3 70 c7 15 ab  ef 94 ec 6c 2e 89 16 d3  |J)r.p......l....|
00000080  ab 66 cf f3 5a 23 98 37  2b c4 77 a5 ab 97 cb 5a  |.f..Z#.7+.w....Z|
00000090  44 ad ac ef 22 1a a7 f4  42 be 91 b9 4d a1 e8 fc  |D..."...B...M...|
000000a0  5b 69 cf 38 5c 3a 71 08  ea f3 d3 58 42 ad 7a 1d  |[i.8\:q....XB.z.|
000000b0  d3 08 a7 ec 7b 24 47 80  30 ee d5 8a 46 f7 6a a3  |....{$G.0...F.j.|
000000c0  14 b8 b8 aa 41 19 21 4e  b6 92 08 c7 97 60 b0 ee  |....A.!N.....`..|
000000d0  4c b5 7c 2b 5a 5e 3e 12  f4 b0 1d a0 a4 ff 12 85  |L.|+Z^>.........|
000000e0  aa 59 d5 e2 56 65 92 58  d7 75 a7 05 c6 7a 8b 39  |.Y..Ve.X.u...z.9|
000000f0  61 bb 3d 5f 94 9e 38 3b  1a 90 4a 58 44 b8 90 21  |a.=_..8;..JXD..!|
00000100  d3 3b 36 bf ec 6e a9 b4  c3 5e f0 36 8b 26 c8 79  |.;6..n...^.6.&.y|
00000110  bf c7 54 b1 64 9f 79 d7  89 d6 77 11 e1 48 14 a8  |..T.d.y...w..H..|
00000120  29 d3 eb 85 fa ca c1 5d  5e 61 ce 8a e5 4c f2 46  |)......]^a...L.F|
00000130  a3 d7 de 68 33 00 00 00  00 00 00 00 00 00 00 00  |...h3...........|
00000140  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000310  00 00 00 00 00                                    |.....|
00000315

Decryption

% picocrypt note_from_windows.txt.pcv
Password: *******
zsh: killed     picocrypt note_from_windows.txt.pcv
% sysctl vm.loadavg #this command ran in another tmux pane while decryption was running
vm.loadavg=2.30 0.80 0.50

Dmesg

% dmesg
...
UVM: pid 40297 (picocrypt), uid 1000 killed: out of swap
UVM: pid 4839 (sshd-session), uid 1000 killed: out of swap
UVM: pid 94147 (picocrypt), uid 1000 killed: out of swap
UVM: pid 15403 (picocrypt), uid 1000 killed: out of swap
UVM: pid 58075 (picocrypt), uid 1000 killed: out of swap
UVM: pid 16298 (picocrypt), uid 1000 killed: out of swap
HACKERALERT commented 14 hours ago

You need at least 2 GiB of RAM available for Picocrypt to run reliably due to Argon2 itself requiring 1 GiB of memory. Looks like you don't have enough in this case: {A8AF21B6-9FD0-478D-B5FF-A0223178D391}

776d commented 13 hours ago

Right. I increased the RAM to 2GiB and it worked encrypting the sample note file. However, I decided to test a somewhat large file as large archives are what I would usually be encrypting (much larger than 670MB as shown here). I increased CPU's to 4 and RAM to 6GiB:

% sysctl hw.ncpu
hw.ncpu=4
% sysctl hw.physmem
hw.physmem=5962747904
% curl -O https://cdn.openbsd.org/pub/OpenBSD/7.6/amd64/install76.iso
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  670M  100  670M    0     0  13.0M      0  0:00:51  0:00:51 --:--:-- 12.0M
% picocrypt install76.iso
Password: ******* | Confirm: *******
Encrypting:  34% [___________________________                                         | (44 MiB/s) [5s:10s]fatal error: runtime: out of memory

runtime stack:
runtime.throw({0x55644d?, 0x0?})
        /usr/local/go/src/runtime/panic.go:1067 +0x4c fp=0x281f4e960 sp=0x281f4e930 pc=0x468b6c
runtime.sysMapOS(0xc05d400000, 0x400000)
        /usr/local/go/src/runtime/mem_bsd.go:81 +0x11b fp=0x281f4e9a0 sp=0x281f4e960 pc=0x4136bb
runtime.sysMap(0xc05d400000, 0x400000, 0x281f4e958?)
        /usr/local/go/src/runtime/mem.go:155 +0x34 fp=0x281f4e9c0 sp=0x281f4e9a0 pc=0x413454
runtime.(*mheap).grow(0x68ddc0, 0x80?)
        /usr/local/go/src/runtime/mheap.go:1539 +0x23b fp=0x281f4ea30 sp=0x281f4e9c0 pc=0x4264db
runtime.(*mheap).allocSpan(0x68ddc0, 0x80, 0x0, 0x1)
        /usr/local/go/src/runtime/mheap.go:1244 +0x2e6 fp=0x281f4ead8 sp=0x281f4ea30 pc=0x425b86
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:962 +0x65 fp=0x281f4eb20 sp=0x281f4ead8 pc=0x425505
runtime.systemstack(0xc000007180)
        /usr/local/go/src/runtime/asm_amd64.s:514 +0x4e fp=0x281f4eb30 sp=0x281f4eb20 pc=0x46e7ae

goroutine 1 gp=0xc0000061c0 m=3 mp=0xc000044e08 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:475 +0x8 fp=0xc000025478 sp=0xc000025468 pc=0x46e748
runtime.(*mheap).alloc(0x100000?, 0x80?, 0x0?)
        /usr/local/go/src/runtime/mheap.go:956 +0x5f fp=0xc0000254c0 sp=0xc000025478 pc=0x42543f
runtime.(*mcache).allocLarge(0x0?, 0x100000, 0x1)
        /usr/local/go/src/runtime/mcache.go:234 +0x87 fp=0xc000025510 sp=0xc0000254c0 pc=0x4124e7
runtime.mallocgc(0x100000, 0x52c120, 0x1)
        /usr/local/go/src/runtime/malloc.go:1177 +0x5d0 fp=0xc0000255b0 sp=0xc000025510 pc=0x464e10
runtime.makeslice(0xc000042088?, 0xc05d280000?, 0x100000?)
        /usr/local/go/src/runtime/slice.go:116 +0x52 fp=0xc0000255d8 sp=0xc0000255b0 pc=0x46a812
main.work()
        /home/user/go/pkg/mod/github.com/!picocrypt/!c!l!i/picocrypt@v0.0.0-20241112002509-12591ffeed33/main.go:496 +0x18c9 fp=0xc000025ed8 sp=0xc0000255d8 pc=0x51c949
main.main()
        /home/user/go/pkg/mod/github.com/!picocrypt/!c!l!i/picocrypt@v0.0.0-20241112002509-12591ffeed33/main.go:675 +0x14e fp=0xc000025f50 sp=0xc000025ed8 pc=0x51e26e
runtime.main()
        /usr/local/go/src/runtime/proc.go:272 +0x28b fp=0xc000025fe0 sp=0xc000025f50 pc=0x435a8b
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc000025fe8 sp=0xc000025fe0 pc=0x470661

goroutine 2 gp=0xc000006700 m=nil [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003efa8 sp=0xc00003ef88 pc=0x468c8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:337 +0xa9 fp=0xc00003efe0 sp=0xc00003efa8 pc=0x435dc9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003efe8 sp=0xc00003efe0 pc=0x470661
created by runtime.init.7 in goroutine 1
        /usr/local/go/src/runtime/proc.go:325 +0x1e

goroutine 3 gp=0xc000006c40 m=nil [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003f780 sp=0xc00003f760 pc=0x468c8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.bgsweep(0xc000058000)
        /usr/local/go/src/runtime/mgcsweep.go:317 +0xe5 fp=0xc00003f7c8 sp=0xc00003f780 pc=0x421d05
runtime.gcenable.gowrap1()
        /usr/local/go/src/runtime/mgc.go:203 +0x25 fp=0xc00003f7e0 sp=0xc00003f7c8 pc=0x416365
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003f7e8 sp=0xc00003f7e0 pc=0x470661
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:203 +0x68

goroutine 4 gp=0xc000006e00 m=nil [GC scavenge wait]:
runtime.gopark(0x10000?, 0x584530?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003ff78 sp=0xc00003ff58 pc=0x468c8e
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:430
runtime.(*scavengerState).park(0x6844e0)
        /usr/local/go/src/runtime/mgcscavenge.go:425 +0x4d fp=0xc00003ffa8 sp=0xc00003ff78 pc=0x41f60d
runtime.bgscavenge(0xc000058000)
        /usr/local/go/src/runtime/mgcscavenge.go:658 +0x65 fp=0xc00003ffc8 sp=0xc00003ffa8 pc=0x41fba5
runtime.gcenable.gowrap2()
        /usr/local/go/src/runtime/mgc.go:204 +0x25 fp=0xc00003ffe0 sp=0xc00003ffc8 pc=0x416305
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003ffe8 sp=0xc00003ffe0 pc=0x470661
created by runtime.gcenable in goroutine 1
        /usr/local/go/src/runtime/mgc.go:204 +0xa5

goroutine 5 gp=0xc000007340 m=nil [finalizer wait]:
runtime.gopark(0x490013?, 0xc00003e660?, 0x1e?, 0xcd?, 0x2abc509a8?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003e620 sp=0xc00003e600 pc=0x468c8e
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:193 +0x10d fp=0xc00003e7e0 sp=0xc00003e620 pc=0x41538d
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003e7e8 sp=0xc00003e7e0 pc=0x470661
created by runtime.createfing in goroutine 1
        /usr/local/go/src/runtime/mfinal.go:163 +0x45

goroutine 6 gp=0xc000007500 m=nil [select, locked to thread]:
runtime.gopark(0xc0000407a8?, 0x2?, 0xb0?, 0x6?, 0xc000040798?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000040638 sp=0xc000040618 pc=0x468c8e
runtime.selectgo(0xc0000407a8, 0xc000040794, 0x0?, 0x0, 0x2?, 0x1)
        /usr/local/go/src/runtime/select.go:335 +0x7a5 fp=0xc000040760 sp=0xc000040638 pc=0x447b85
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:1060 +0x171 fp=0xc0000407e0 sp=0xc000040760 pc=0x461eb1
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc0000407e8 sp=0xc0000407e0 pc=0x470661
created by runtime.ensureSigM in goroutine 1
        /usr/local/go/src/runtime/signal_unix.go:1043 +0xc8

goroutine 7 gp=0xc000007a40 m=2 mp=0xc000044708 [syscall]:
runtime.notetsleepg(0x6a4e20, 0xffffffffffffffff)
        /usr/local/go/src/runtime/lock_sema.go:296 +0x2d fp=0xc000040fa0 sp=0xc000040f68 pc=0x40bf0d
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2d fp=0xc000040fc0 sp=0xc000040fa0 pc=0x46a60d
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x17 fp=0xc000040fe0 sp=0xc000040fc0 pc=0x4d3f17
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc000040fe8 sp=0xc000040fe0 pc=0x470661
created by os/signal.Notify.func1.1 in goroutine 1
        /usr/local/go/src/os/signal/signal.go:151 +0x25

goroutine 8 gp=0xc000007c00 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc0000416a8 sp=0xc000041688 pc=0x468c8e
runtime.chanrecv(0xc000106070, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:639 +0x41c fp=0xc000041720 sp=0xc0000416a8 pc=0x4067bc
runtime.chanrecv1(0x0?, 0x0?)
        /usr/local/go/src/runtime/chan.go:489 +0x16 fp=0xc000041748 sp=0xc000041720 pc=0x406396
main.main.func1()
        /home/user/go/pkg/mod/github.com/!picocrypt/!c!l!i/picocrypt@v0.0.0-20241112002509-12591ffeed33/main.go:641 +0x29 fp=0xc0000417e0 sp=0xc000041748 pc=0x51e3a9
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc0000417e8 sp=0xc0000417e0 pc=0x470661
created by main.main in goroutine 1
        /home/user/go/pkg/mod/github.com/!picocrypt/!c!l!i/picocrypt@v0.0.0-20241112002509-12591ffeed33/main.go:640 +0xe7

goroutine 9 gp=0xc000007dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc000041f38 sp=0xc000041f18 pc=0x468c8e
runtime.gcBgMarkWorker(0xc000068230)
        /usr/local/go/src/runtime/mgc.go:1363 +0xef fp=0xc000041fc8 sp=0xc000041f38 pc=0x41848f
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc000041fe0 sp=0xc000041fc8 pc=0x418365
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc000041fe8 sp=0xc000041fe0 pc=0x470661
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 10 gp=0xc00012a000 m=nil [GC worker (idle)]:
runtime.gopark(0xc00003a7a0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003a738 sp=0xc00003a718 pc=0x468c8e
runtime.gcBgMarkWorker(0xc000068230)
        /usr/local/go/src/runtime/mgc.go:1363 +0xef fp=0xc00003a7c8 sp=0xc00003a738 pc=0x41848f
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00003a7e0 sp=0xc00003a7c8 pc=0x418365
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003a7e8 sp=0xc00003a7e0 pc=0x470661
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 11 gp=0xc00012a1c0 m=nil [GC worker (idle)]:
runtime.gopark(0xc00003afa0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003af38 sp=0xc00003af18 pc=0x468c8e
runtime.gcBgMarkWorker(0xc000068230)
        /usr/local/go/src/runtime/mgc.go:1363 +0xef fp=0xc00003afc8 sp=0xc00003af38 pc=0x41848f
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00003afe0 sp=0xc00003afc8 pc=0x418365
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003afe8 sp=0xc00003afe0 pc=0x470661
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105

goroutine 12 gp=0xc00012a380 m=nil [GC worker (idle)]:
runtime.gopark(0xc00003b7a0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:424 +0xce fp=0xc00003b738 sp=0xc00003b718 pc=0x468c8e
runtime.gcBgMarkWorker(0xc000068230)
        /usr/local/go/src/runtime/mgc.go:1363 +0xef fp=0xc00003b7c8 sp=0xc00003b738 pc=0x41848f
runtime.gcBgMarkStartWorkers.gowrap1()
        /usr/local/go/src/runtime/mgc.go:1279 +0x25 fp=0xc00003b7e0 sp=0xc00003b7c8 pc=0x418365
runtime.goexit({})
        /usr/local/go/src/runtime/asm_amd64.s:1699 +0x1 fp=0xc00003b7e8 sp=0xc00003b7e0 pc=0x470661
created by runtime.gcBgMarkStartWorkers in goroutine 1
        /usr/local/go/src/runtime/mgc.go:1279 +0x105
HACKERALERT commented 11 hours ago

Hmm that's odd. The file size shouldn't matter as the CLI will read/encrypt/write in 1 MiB chunks so the memory consumption during encryption should be constant regardless of input file size. Maybe the Go garbage collector just isn't working well in this case 🤷 . Can you find the "breaking point" of file size? like the maximum file size you can encrypt without getting an error. That may be a hint.

If you're familiar with go, you can also can add a debug.FreeOSMemory (doc) to https://github.com/Picocrypt/CLI/blob/main/picocrypt/main.go#L586 and go run main.go install76.iso but otherwise I'm not sure what's happening.

HACKERALERT commented 11 hours ago

If you don't know or want to install Go, ping me and I'll add that line and trigger a new release. I know it's in the debug package, but it's generally safe to use in production at a slight cost of performance (but probably marginal).

776d commented 10 hours ago

Hmm that's odd. The file size shouldn't matter as the CLI will read/encrypt/write in 1 MiB chunks so the memory consumption during encryption should be constant regardless of input file size. Maybe the Go garbage collector just isn't working well in this case 🤷 . Can you find the "breaking point" of file size? like the maximum file size you can encrypt without getting an error. That may be a hint.

I will try this later and report back.

If you're familiar with go, you can also can add a debug.FreeOSMemory (doc) to https://github.com/Picocrypt/CLI/blob/main/picocrypt/main.go#L586 and go run main.go install76.iso but otherwise I'm not sure what's happening.

I did as you said here and imported runtime/debug and it actually encrypts the install76.iso file successfully. I will test an even larger file later and see if it works. If so, that’s all I really care about. Performance doesn’t matter to me as long as it isn’t abysmally slow. The time to encrypt the example install iso was decent anyways.

HACKERALERT commented 9 hours ago

Awesome, I've pushed a commit and made a new release that adds the debug.FreeOSMemory() in the three locations where necessary. You should be able to use without error now:

go install github.com/Picocrypt/CLI/picocrypt@7a8ce46