Segmentation fault with v2.2

[mihails-strasuns]

When testing my digger package, got this:

$ cat /etc/digger.init
workDir = ~/.cache/digger/
cache = git
[environment]
$ digger build --bootstrap v2.066.1
# ...
digger: Saving to cache.
digger: Installing phobos-includes-c8fccac8c20a3bdd6300128f610267a24d42473f-e6520d80c9857e15cf978a6607bf09d1
Segmentation fault (core dumped)

backtrace:

#0  0x00007f7406cfd4f8 in __memset_avx2 () from /usr/lib/libc.so.6
#1  0x0000000000760209 in _d_newarrayT ()
#2  0x000000000074c9e3 in ae.sys.git.Repository.ObjectReaderImpl.read(immutable(char)[]) ()
#3  0x000000000074cbc2 in ae.sys.git.Repository.ObjectReaderImpl.read(ubyte[20]) ()
#4  0x000000000074e2be in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv ()
#5  0x000000000074e38b in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv ()
#6  0x000000000074e38b in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv ()
#7  0x000000000074e010 in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZv ()
#8  0x000000000074df7b in _D2ae3sys3git10Repository12exportCommitMFAyaAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZv ()
#9  0x00000000006ff2b0 in ae.sys.d.cache.GitCache.extract(immutable(char)[], immutable(char)[], bool(immutable(char)[]) delegate) ()
#10 0x00000000006e7e39 in ae.sys.d.manager.DManager.Component.install() ()
#11 0x00000000006e7cad in ae.sys.d.manager.DManager.Component.needInstalled() ()
#12 0x00000000006e75b5 in ae.sys.d.manager.DManager.Component.needBuild() ()
#13 0x00000000006e79e9 in ae.sys.d.manager.DManager.Component.needInstalled() ()
#14 0x00000000006eaa1a in ae.sys.d.manager.DManager.build(ae.sys.d.manager.DManager.SubmoduleState, ae.sys.d.manager.DManager.Config.Build, bool) ()
#15 0x0000000000727092 in custom.runBuild(immutable(char)[], ae.sys.d.manager.DManager.SubmoduleState, repo.BuildConfig) ()
#16 0x000000000072884c in custom.buildCustom(immutable(char)[], repo.BuildConfig) ()
#17 0x0000000000679815 in digger.Digger.build(ae.utils.funopt.OptionImpl!(0, bool, "hiddenOption", '\x00', null, "64").OptionImpl, ae.utils.funopt.OptionImpl!(1, immutable(char)[], "Select model (32 or 64). On this system, the default is 64", '\x00', null, "model").OptionImpl, ae.utils.funopt.OptionImpl!(1, immutable(char)[][], "Additional make parameters, e.g. "-j8" or "HOST_CC=g++48"", '\x00', "ARG", "makeArgs").OptionImpl, ae.utils.funopt.OptionImpl!(0, bool, "Bootstrap the compiler (build from C++ source code) instead of downloading a pre-built binary package", '\x00', null, "bootstrap").OptionImpl, ae.utils.funopt.OptionImpl!(2, immutable(char)[], "D ref (branch / tag / point in time) to build, plus any additional forks or pull requests. Example:
"master @ 3 weeks ago + dmd#123 + You/dmd/awesome-feature"", '\x00', null, null).OptionImpl) ()
#18 0x00000000006d329d in _D2ae5utils6funopt1705__T6funoptS1390_D6digger6Digger5buildFS2ae5utils6funopt102__T10OptionImplVE2ae5utils6funopt10OptionTypei0TbVAyaa12_68696464656e4f7074696f6eVai0VAyanVAyaa2_3634Z10OptionImplS2ae5utils6funopt202__T10OptionImplVE2ae5utils6funopt10OptionTypei1TAyaVAyaa58_53656c656374206d6f64656c20283332206f72203634292e204f6e20746869732073797374656d2c207468652064656661756c74206973203634Vai0VAyanVAyaa5_6d6f64656cZ10OptionImplS2ae5utils6funopt215__T10OptionImplVE2ae5utils6funopt10OptionTypei1TAAyaVAyaa57_4164646974696f6e616c206d616b6520706172616d65746572732c20652e672e20222d6a3822206f722022484f53545f43433d672b2b343822Vai0VAyaa3_415247VAyaa8_6d616b6541726773Z10OptionImplS2ae5utils6funopt295__T10OptionImplVE2ae5utils6funopt10OptionTypei0TbVAyaa101_426f6f7473747261702074686520636f6d70696c657220286275696c642066726f6d20432b2b20736f7572636520636f64652920696e7374656164206f6620646f776e6c6f6164696e672061207072652d6275696c742062696e617279207061636b616765Vai0VAyanVAyaa9_626f6f747374726170Z10OptionImplS2ae5utils6funopt391__T10OptionImplVE2ae5utils6funopt10OptionTypei2TAyaVAyaa158_442072656620286272616e6368202f20746167202f20706f696e7420696e2074696d652920746f206275696c642c20706c757320616e79206164646974696f6e616c20666f726b73206f722070756c6c2072657175657374732e204578616d706c653a0a226d617374657220402033207765656b732061676f202b20646d6423313233202b20596f752f646d642f617765736f6d652d6665617475726522Vai0VAyanVAyanZ10OptionImplZiVS2ae5utils6funopt12FunOptConfigS1nS260_D2ae5utils6funopt107__T14funoptDispatchTS6digger6DiggerVS2ae5utils6funopt12FunOptConfigS1nS33_D6digger6diggerFZ8usageFunFAyaZvZ14funoptDispatchFAAyaZ3funMFAyaAAyaZ74__T12descUsageFunVAyaa24_4275696c6420442066726f6d20736f7572636520636f6465Z12descUsageFunFAyaZvZ6funoptFAAyaZi ()
#19 0x00000000006d20e8 in _D2ae5utils6funopt107__T14funoptDispatchTS6digger6DiggerVS2ae5utils6funopt12FunOptConfigS1nS33_D6digger6diggerFZ8usageFunFAyaZvZ14funoptDispatchFAAyaZ3funMFAyaAAyaZi ()
#20 0x00000000006e1778 in _D2ae5utils6funopt390__T6funoptS165_D2ae5utils6funopt107__T14funoptDispatchTS6digger6DiggerVS2ae5utils6funopt12FunOptConfigS1nS33_D6digger6diggerFZ8usageFunFAyaZvZ14funoptDispatchFAAyaZ3funMFAyaAAyaZiVS2ae5utils6funopt12FunOptConfigS1A1i6S168_D2ae5utils6funopt107__T14funoptDispatchTS6digger6DiggerVS2ae5utils6funopt12FunOptConfigS1nS33_D6digger6diggerFZ8usageFunFAyaZvZ14funoptDispatchFAAyaZ10myUsageFunFAyaZvZ6funoptMFAAyaZi ()
#21 0x00000000006d1f50 in _D2ae5utils6funopt107__T14funoptDispatchTS6digger6DiggerVS2ae5utils6funopt12FunOptConfigS1nS33_D6digger6diggerFZ8usageFunFAyaZvZ14funoptDispatchFAAyaZi ()
#22 0x0000000000679eb2 in digger.digger() ()
#23 0x000000000067a185 in digger.__mixin19.run(immutable(char)[][]) ()
#24 0x000000000067a1b1 in D main ()

Running same command again without cleaning the cache works fine.

[CyberShadow]

Can you reproduce this reliably?

[mihails-strasuns]

2 hits in 2 attempts :)

[CyberShadow]

As far as I understand, this can happen if Git reports an enormous (or negative) object size, and Digger then attempts to allocate RAM for it. Can you add std.stdio.stderr.writeln(header) at line 151 of ae.sys.git OSLT?

[CyberShadow]

Also if you built Digger with DMD, can you try building it with a different compiler? I ran into some codegen issues with DMD with that code. Never mind, IIRC the bug was in DMD-FE, so the other compilers would have it too... DMD HEAD might be worth trying. Also don't use -O, -inline or -release.

[CyberShadow]

Can't reproduce this on Ubuntu Server x64

[mihails-strasuns]

Sorry, I am a bit overwhelmed right now. Will report update as soon as I will get back to digger packaging.

[CyberShadow]

Might also be worth testing with D master or D master + https://github.com/D-Programming-Language/phobos/pull/2794

[mihails-strasuns]

still can reproduce it in 100% of cases

output of your suggested print statement:

["0d4608901d78a6faaa909f4ad7cab99bf6068f33", "commit", "271"]
["5e465ba653e942d620256e27fe115d42d4ba584c", "tree", "127"]
["6dfe8c8dbdd3ac2e5f2c68471a9095aa54a1f306", "tree", "60"]
["cfcfb0e015b5cededcba3829c7adbfc23312fd31", "tree", "28"]
["6db3954652c13557dfc273190b449c7be55acedd", "tree", "136"]
["f4a6bb0e810a5fa89c10829099ec03b19a2c4f46", "blob", "82291"]
["b4d9f78cfb7ec6560a14c1a10bfb54ffec70ffd3", "blob", "50961"]
["1aec625cdd4c3661093f7c895f854af6cf26a2a3", "tree", "1312"]
["f310bb0fcdb176111b1ded729adf39484a0e9551", "blob", "62259"]
["d4219bf889f5bb7e8bc70ea7d092a9974dcbcf8d", "blob", "5200"]
["65ad6a5adc44c00428b35eb15ff26d16f9ef209b", "blob", "5048"]
["34960bddaccdb7ab4cf0a8a0f4370137c8a110e5", "blob", "9336"]
["ea4dfbe9d7b3ee2b7b380cc435973aca1976d4c0", "blob", "2520"]
["91be372d224da6833c03bd24d7158ff622ece35f", "blob", "13681"]
["8053b6117c023f64554ea783b23fa06985acff7b", "blob", "30568"]
["5c4022f3d4774b98bb22104890b04a7d3f7f4aec", "blob", "67992"]
["cbf0d1ea5d966e572c4d3ae24d334c1ff1789169", "blob", "12672"]
["604736f15f67ba668f20c17c7ac41c27d06c9bf0", "blob", "16398"]
["caeb99a3177f477d622870255a00ac2b72f10cad", "blob", "678"]
["0f8fb79f87d4f65e7f5667129c763e3a361199f2", "blob", "4677"]
["603e60ed5444e0e7a2ed9590a9d9a73ce7cc97bf", "blob", "14071"]
["548201ab0099b51556da32c4873d1dbb2d211b5a", "blob", "20574"]
["e8defc6887a186055f17597968c4fc5791332811", "blob", "14602"]
["af3a8c965d50cc6ac05c6c844a8f79e2cdc9f3c1", "blob", "22622"]
["2f1d60b43b8a9deacf22bb5526a979dcc219a68f", "blob", "13439"]
["e5c1aa4ca8cd5244423680865609c71ab68f9ab6", "blob", "427"]
["75ed4b5978de4be3c44ad48a060b75500f17a0a2", "blob", "6343"]
["a8431abeacffd97e5028a9e54135a5f173528960", "blob", "52623"]
["95f4986d400223bad542e5b34a7e6284a039425e", "blob", "6399"]
["11e9c52accbe8e2cf4e502f88ce89ccfa4f992e5", "blob", "13769"]
["baa53a0b1a199ce6ea4c3f99d0306502ab4fab2c", "blob", "2928"]
["604f70027c6ae4558e1833d36fdc868b94974de1", "blob", "2004"]
["9825ccc3a712e19fe8bccb52214f9da823a6757f", "blob", "11216"]
["604f70027c6ae4558e1833d36fdc868b94974de1", "blob", "2004"]
["56e9bb1c115e81082f58a89b003b766760e12a12", "blob", "45242"]
["d35639d82a27807e49ea35c334f8bbcf64720f82", "blob", "8472"]
["ad98be3a5d88afd479d3442cd234747ca259c60d", "blob", "1994"]
["f03a4b8f66738b241036cd2bc72b83276311bfc0", "blob", "2004"]
["1edfd0e65453a81e7f83bf20bb7ee3cd7739619e", "blob", "2211"]
["02ce56c4313b7e7490c0416ed52bfd9c65694553", "blob", "13375"]
["27adc4cd10a35713943497e5c4c91b09e70e74ef", "blob", "4235"]
["bfbba83e8ee0902abe8bcbfab8bdebbbc07dca69", "blob", "79564"]
["898ed345b0e3ef23ce5aaba1eb6307b5b113d6b1", "blob", "7313"]
["258fa88799ad1d97e44fe1ac9df77948e82ea5c0", "blob", "7153"]
["65b1d923122534e90cf79bd32cb777aa6f4711d8", "blob", "63321"]
["eb5b48150e83e92e070482b6b0dc04507d00fb7f", "tree", "2245"]
["236291e182598cf2f4df9956e15c94107424da74", "blob", "393076"]
["7d1fb871e1d4faf82f7058068dab0607984062ff", "blob", "95939"]
["493fd6f6ee57636617abf5766ebdf7db62518c07", "blob", "11339"]
["270bd2c73682cc58485470efd30d963695ab0a32", "blob", "52292"]
["ef568a219d3a681e950ca5eabe90f4b20ecf3f9e", "blob", "27212"]
["6d08a641bdf6f94eb1cc4db3444aed3d24262bee", "blob", "96352"]
["c0de5f8d64a7959e7c90199bd156fc0a2b35540f", "tree", "520"]
["02344b5fd70d2cefcd9528902a7ef3a42a80179c", "blob", "216"]
["d6843f5ee12f36389e26e22082ec0160d560dd38", "tree", "36"]
["c75dfd23cc61e7865070c363b00d00d808d875a8", "blob", "841"]
["5e80e5ddccccfe272ff29041b3fac89a3a4b3bfd", "tree", "220"]
["0252f470027eeb8ae1e7677ed49e65ae15f8d3f1", "blob", "1725"]
["8976005cd7846b04f724417d7f3c061d9cdad29f", "blob", "413"]
["959428d7bd9ccaa6afa7092bc8c68e12ff451583", "blob", "234"]
["6d4e34784bdcb0a0a9382471ccbc3a4cbff9c648", "blob", "2880"]
["30e9443936ed97f94c7248bc035cc23117bb10ae", "blob", "82"]
["b2e60e388f3c4f3bf1418bf324c3b493807014d6", "blob", "3705"]
["89d4004e8e5f7d20b8ab4f7f9b34bfa9149fd461", "blob", "199"]
["cc6454a584886b3fd182bd6f47327aebdcb9d87c", "blob", "208"]
["899af2a78995e291e561fb61ba597738f8d26a85", "tree", "36"]
["2b1439fff25daa6056887b63e8551ac0bfb0fda2", "blob", "1967"]
["650b50426efa0695367796c17bc5413f11c77835", "blob", "2473"]
["8fd2d4915a3192b6a670d71db7b5bd34f3d76618", "blob", "296"]
["8f41d5d35bf663b0f3cb7c4a25b659026380b108", "blob", "224"]
["40ea1b228563f56d67c2edded2fdf47ac69682ba", "blob", "334"]
["3b34a9e174f835bb493a9a6359176197cc30bd4e", "blob", "478"]
["8382e63b9dd0541948efa7929b5c91a23486f7b5", "blob", "559"]
["f0fe47a62c9c3c044fe9f02ddc1c0725a415efac", "blob", "209"]
["f4e93cc42c7d0e9ef911f8104d404a49ae2ce19a", "blob", "215"]
["9fd2499f65f7bc260ab9428aaf357c3cf208219d", "tree", "141"]
["f862e1025d0c6dfc04350713d003f2afaf6a4972", "blob", "6998"]
["1d219e1eec62ed4c2b8eed73aae1640c37004b92", "blob", "1041"]
["8e44ed07f8d8e0d50f3c2d8de88278e053980bae", "blob", "199"]
["6f65ff0960dbfa04ffc1d6d9939bead10ca73e52", "blob", "15515"]
["b469bffb541328cf18c93e236b6d622b5dbb7b4c", "blob", "1975"]
["d3ba37c80091631eabda6b4067cfc0b8ca459434", "blob", "22346"]
["abcc8c197219473c667956e201caba4a096a6eb1", "blob", "45673"]
["ec0527e8d2504e87b70b1f89776a3999f4838192", "tree", "252"]
["1ed719de0fea7d5c8f1e6345332ff2cad95b6791", "blob", "51077"]
["78db6d2b45fc6d635d1ee9484780db635dccaf97", "blob", "13216"]
["2cf7ba2c902410a4d9c26491182cff3e99cff78d", "blob", "19467"]
["5cc47d671fa7359dc9f96e474815b23e8c72f8e2", "blob", "23451"]
["540a7576b3dbc77ad380671a6b9f7c8157769386", "blob", "48850"]
["d94cb76a81d6c2b7d76cbf8ee7042b80cc300ec1", "blob", "15004"]
["5c9e5a83f762810e56746b61c4a21b0b52ff9532", "blob", "2086"]
["8ce8ad6eef2709e38c10c12460b0a04972770987", "blob", "137808"]
["732245740ad667f107471f7d6781c0960aa89c9a", "blob", "6220"]
["da4a032a606244d35e09a868671b66e45b1e27eb", "blob", "45118"]
["1b079a4d3e25ced875c1f00f37d3009e15113ee6", "blob", "1504695"]

backtrace head with line numbers:

#0  0x00007f445f60e4f8 in __memset_avx2 () from /usr/lib/libc.so.6
(gdb) bt
#0  0x00007f445f60e4f8 in __memset_avx2 () from /usr/lib/libc.so.6
#1  0x000000000076db39 in _d_newarrayT ()
#2  0x000000000075996b in ae.sys.git.Repository.ObjectReaderImpl.read(immutable(char)[]) (this=0xc8ae90, __HID415=0x7ffd26d56b70, name=...) at ae/sys/git.d:161
#3  0x0000000000759b8a in ae.sys.git.Repository.ObjectReaderImpl.read(ubyte[20]) (this=0xc8ae90, __HID416=0x7ffd26d56b70, hash=...) at ae/sys/git.d:176
#4  0x000000000075b3d9 in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv (this=0x7ffd26d56fe0, subPath=..., treeHash=...) at ae/sys/git.d:317
#5  0x000000000075b4a8 in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv (this=0x7ffd26d56fe0, subPath=..., treeHash=...) at ae/sys/git.d:327
#6  0x000000000075b4a8 in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZ13exportSubTreeMFG20hAAyaZv (this=0x7ffd26d56fe0, subPath=..., treeHash=...) at ae/sys/git.d:327
#7  0x000000000075b0e3 in _D2ae3sys3git10Repository10exportTreeMFG20hAyaS3std8typecons102__T10RefCountedTS2ae3sys3git10Repository16ObjectReaderImplVE3std8typecons24RefCountedAutoInitializei1Z10RefCountedDFAyaZbZv (this=0x7f4460051828, pathFilter=..., reader=..., path=..., treeHash=...) at ae/sys/git.d:337

binary is built with rdmd --build-only -g digger.d

[mihails-strasuns]

No crash with DMD/Phobos master

[CyberShadow]

It's probably the readln memory corruption bug, then. Suggested fix: build with D master.

[CyberShadow]

Or try with Digger master again, I worked around it by implementing my own readln

[mihails-strasuns]

Still crashes with digger master on 2.067 :(

[CyberShadow]

Did you remember to update the submodules?

If yes, try this patch: http://dump.thecybershadow.net/3c1ad60daffa2658321a0c3b7260612d/000000BB.txt

[mihails-strasuns]

yes, linked patch fixes it

[CyberShadow]

Got it, appender sux. Digger master updated

[mihails-strasuns]

any chance for 2.2.1 tag?

[CyberShadow]

Nope, 2.3

[mihails-strasuns]

thanks!