search

almostEric / FrozenWasteland

Plugins for VCV Rack
Other
91 stars 18 forks source link

v1: Portland Weather SEGFAULT #30

Closed cschol closed 5 years ago

cschol commented 5 years ago

Portland Weather segfaults in v1 on Ubuntu 18.04:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:371
371     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
[Current thread is 1 (Thread 0x7efd9ffff700 (LWP 15698))]
(gdb) bt
#0  0x00007efdbabbbc40 in __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:371
#1  0x00007efdb12f2983 in memcpy (__len=<optimized out>, __src=0x7efd97ffd2f0, __dest=0x7efd97ffd2b0) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#2  0x00007efdb12f2983 in FrozenWasteland::DoubleRingBuffer<float, 16ul>::endIncr(unsigned long) (n=139628224306854, this=0x7efd97ffd2b0) at ./src/dsp-delay/ringbuffer.hpp:120
#3  0x00007efdb12f2983 in PortlandWeather::process(rack::engine::Module::ProcessArgs const&) (this=0x7efd8bdfa010, args=...) at src/PortlandWeather.cpp:564
#4  0x0000555ea515416f in rack::engine::Engine_stepModules(rack::engine::Engine*, int) (that=that@entry=0x555ea65130f0, threadId=0) at src/engine/Engine.cpp:248
#5  0x0000555ea5156d09 in rack::engine::Engine_step (that=0x555ea65130f0) at src/engine/Engine.cpp:298
#6  0x0000555ea5156d09 in rack::engine::Engine_run (that=0x555ea65130f0) at src/engine/Engine.cpp:409
#7  0x0000555ea5156d09 in rack::engine::Engine::<lambda()>::operator() (__closure=<optimized out>) at src/engine/Engine.cpp:438
#8  0x0000555ea5156d09 in std::__invoke_impl<void, rack::engine::Engine::start()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#9  0x0000555ea5156d09 in std::__invoke<rack::engine::Engine::start()::<lambda()> > (__fn=...) at /usr/include/c++/7/bits/invoke.h:95
#10 0x0000555ea5156d09 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /usr/include/c++/7/thread:234
#11 0x0000555ea5156d09 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::operator() (this=<optimized out>) at /usr/include/c++/7/thread:243
#12 0x0000555ea5156d09 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > > >::_M_run(void) (this=<optimized out>) at /usr/include/c++/7/thread:186
#13 0x00007efdbb49166f in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#14 0x00007efdbcde06db in start_thread (arg=0x7efd9ffff700) at pthread_create.c:463
#15 0x00007efdbab4e88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
almostEric commented 5 years ago

Hi Chris,

The error where this code is occurring is directly copied from the fundamental delay module (just with buffers arrays made multidimensional to handle the multiple taps). And the ringbuffer code (line 120) is identical to the fundamental ringbuffer (I’ve added a MultiTapRingBuffer but that is not where this is crashing).

Is there some compiler directive I should be adding to my MakeFile? I see the error, but not really sure what is causing it

Eric

From: cschol notifications@github.com Sent: Friday, May 24, 2019 3:32 PM To: almostEric/FrozenWasteland FrozenWasteland@noreply.github.com Cc: Subscribed subscribed@noreply.github.com Subject: [almostEric/FrozenWasteland] v1: Portland Weather SEGFAULT (#30)

Portland Weather segfaults in v1 on Ubuntu 18.04:

Program terminated with signal SIGSEGV, Segmentation fault.

0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:371

371 ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory. [Current thread is 1 (Thread 0x7efd9ffff700 (LWP 15698))] (gdb) bt

0 0x00007efdbabbbc40 in __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:371

1 0x00007efdb12f2983 in memcpy (len=, src=0x7efd97ffd2f0, __dest=0x7efd97ffd2b0) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34

2 0x00007efdb12f2983 in FrozenWasteland::DoubleRingBuffer<float, 16ul>::endIncr(unsigned long) (n=139628224306854, this=0x7efd97ffd2b0) at ./src/dsp-delay/ringbuffer.hpp:120

3 0x00007efdb12f2983 in PortlandWeather::process(rack::engine::Module::ProcessArgs const&) (this=0x7efd8bdfa010, args=...) at src/PortlandWeather.cpp:564

4 0x0000555ea515416f in rack::engine::Engine_stepModules(rack::engine::Engine*, int) (that=that@entry=0x555ea65130f0, threadId=0) at src/engine/Engine.cpp:248

5 0x0000555ea5156d09 in rack::engine::Engine_step (that=0x555ea65130f0) at src/engine/Engine.cpp:298

6 0x0000555ea5156d09 in rack::engine::Engine_run (that=0x555ea65130f0) at src/engine/Engine.cpp:409

7 0x0000555ea5156d09 in rack::engine::Engine::<lambda()>::operator() (__closure=) at src/engine/Engine.cpp:438

8 0x0000555ea5156d09 in std::__invoke_impl<void, rack::engine::Engine::start()::<lambda()> > (__f=...) at /usr/include/c++/7/bits/invoke.h:60

9 0x0000555ea5156d09 in std::invoke<rack::engine::Engine::start()::<lambda()> > (fn=...) at /usr/include/c++/7/bits/invoke.h:95

10 0x0000555ea5156d09 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::_M_invoke<0> (this=) at /usr/include/c++/7/thread:234

11 0x0000555ea5156d09 in std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > >::operator() (this=) at /usr/include/c++/7/thread:243

12 0x0000555ea5156d09 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<rack::engine::Engine::start()::<lambda()> > > >::_M_run(void) (this=) at /usr/include/c++/7/thread:186

13 0x00007efdbb49166f in () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007efdbcde06db in start_thread (arg=0x7efd9ffff700) at pthread_create.c:463

15 0x00007efdbab4e88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/almostEric/FrozenWasteland/issues/30?email_source=notifications&email_token=AADPVT5IRL6N33KNYJ4DMCDPW7U45A5CNFSM4HPPSDLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVWSTYA , or mute the thread https://github.com/notifications/unsubscribe-auth/AADPVT2KOGZE2XTIR7PVXOTPW7U45ANCNFSM4HPPSDLA . https://github.com/notifications/beacon/AADPVT7QP5FCHA7GLOWVYYLPW7U45A5CNFSM4HPPSDLKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GVWSTYA.gif

fractalgee commented 5 years ago

../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory...

Could it be the .S instead of .s that causes this? That doesn't look right for an extension

almostEric commented 5 years ago

I don’t know that is not my code. I wonder if it has to do with the std namespace. The .6 pw borrowed it’s delay from fundamental and 1.0 does the same, but it appears a lot has changed under the hood. My guess is for thread safety. If some folks with more experience with some of these libraries could take a look, I’d appreciate it.

I do know I was getting a lot of weird audio artifacts , so I’m sure there’s something weird happening

almostEric commented 5 years ago

added specific reference to the common.hpp code that has some SIMD stuff... we'll see.

AndrewBelt commented 5 years ago

You've accidentally committed the dep/ folder to git, binaries and all. Building object files and linking them on different machines will mostly certainly not work in general. Remove those (and remove the git history to reduce the repo download size), and rebuild.

cschol commented 5 years ago

@almostEric Also, I get notifications for this issue, you don't need to resubmit to the library thread until the issues are resolved. Thanks.

cschol commented 5 years ago

This latest commit (removing the dep/ and rebuilding the library during build) fixes the crash in Portland Weather. However, you need to remove the files from repository history also as Andrew requested above.

The repository history still contains the files and the repository is over 70MB in size still.

Check out this link for more information on how to do that.

almostEric commented 5 years ago

I’m a little nervous using either BFG or git-filter-branch. Can you explain why this needs to be done? I understand there are now some large files in my history, but how does that affect the vcv build process? Wouldn’t you only be downloading the latest commit that doesn’t have the large files anymore?

cschol commented 5 years ago

That's not how git works (as opposed to, say, SVN). When you clone the repository you always get the entire history. Just removing the files in a commit does not solve the repository size issue.

There is no reason to be nervous. Clone your repository into a new folder, perform the cleanup on that local copy, check that everything is correct, and if it is not delete the copy and start over. Only push to Github if you are sure that everything is correct.

cschol commented 5 years ago

Here is what you should need to do (be sure to keep a backup of your repository around!):

  1. Get bfg repo cleaner
cd /tmp
wget http://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar
  1. Clone a mirror of your repository to /tmp (note, the --mirror flag; this will create a bare repository, i.e. no working copy)
cd /tmp
git clone --mirror https://github.com/almostEric/FrozenWasteland.git
  1. Clean offending files from repository
java -jar /tmp/bfg-1.13.0.jar --delete-folders 'dep' FrozenWasteland.git
cd FrozenWasteland.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
  1. Clone working copy from bare repository and check content
cd /tmp
git clone FrozenWasteland.git FrozenWasteland_cleaned
cd FrozenWasteland_cleaned

Check content of cleaned repository. Repository size should be around 12MB in this state (I tried it).

  1. Push cleaned repository to GitHub
git push origin master --force
almostEric commented 5 years ago

Hi,

I seem to be missing something here.

The bfg cleanup seems to do its thing

I do the git reflog.. etc

When I clone the repository, the original is 24MB, the cleaned is 50MB.

If I do a push, it says everything is up to date, nothing changes on github

Eric

From: cschol notifications@github.com Sent: Saturday, May 25, 2019 4:36 PM To: almostEric/FrozenWasteland FrozenWasteland@noreply.github.com Cc: Eric Sterling eric@digitalpour.com; Mention mention@noreply.github.com Subject: Re: [almostEric/FrozenWasteland] v1: Portland Weather SEGFAULT (#30)

Here is what you should need to do (be sure to keep a backup of your repository around!):

  1. Get bfg repo cleaner

cd /tmp wget http://repo1.maven.org/maven2/com/madgag/bfg/1.13.0/bfg-1.13.0.jar

  1. Clone a mirror of your repository to /tmp (note, the --mirror flag; this will create a bare repository, i.e. no working copy)

cd /tmp git clone --mirror https://github.com/almostEric/FrozenWasteland.git

  1. Clean offending files from repository

java -jar /tmp/bfg-1.13.0.jar --delete-folders 'dep' FrozenWasteland.git cd FrozenWasteland.git git reflog expire --expire=now --all && git gc --prune=now --aggressive

  1. Clone working copy from bare repository and check content

cd /tmp git clone FrozenWasteland.git FrozenWasteland_cleaned cd FrozenWasteland_cleaned

Check content of cleaned repository. Repository size should be around 12MB in this state (I tried it).

  1. Push cleaned repository to GitHub

git push origin master --force

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/almostEric/FrozenWasteland/issues/30?email_source=notifications&email_token=AADPVTZ5Q6GKXG7QRZWISF3PXFFEPA5CNFSM4HPPSDLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWHS62Q#issuecomment-495923050 , or mute the thread https://github.com/notifications/unsubscribe-auth/AADPVT6MJC43GALIC6RJZSLPXFFEPANCNFSM4HPPSDLA . https://github.com/notifications/beacon/AADPVT6PUEFX45XTWBKDDETPXFFEPA5CNFSM4HPPSDLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWHS62Q.gif

cschol commented 5 years ago

The "original" is a bare repository, i.e. it does not have any working copy. That's why it is smaller. When you clone your "cleaned" version you clone it from the local bare repository. So, when you do a push in the cleaned repository you push to the local bare repo, which obviously has nothing to update.

Therefore, you need to go to the bare repository, which is referencing Github as an origin and push from there. Or you need to set Github as a remote in the cleaned repository and push then.

almostEric commented 5 years ago

I've gone through these steps several times. I think I see what may be the issue. BFG is aborting, saying "No refs to update - no dirty commits found?"

cschol commented 5 years ago

Did you clone the original (pre-BFG) repo with the --mirror argument? It must be a bare repo for this to work.

almostEric commented 5 years ago

yes. I am literally cutting an pasting from your post

cschol commented 5 years ago

I tried it again and it removes the dep directory for me.

Can you post the output of BFG?

almostEric commented 5 years ago

Sure:

Found 855 objects to protect

Found 29 commit-pointing refs : HEAD, refs/heads/master, refs/heads/v1, ….

Protected commits

These are your protected commits, and so their contents will NOT be altered:

Cleaning

Found 163 commits

Cleaning commits: 100% (163/163)

Cleaning commits in 166ms

BFG aborting: No refs to update – no dirty commits found?

cschol commented 5 years ago

One more question: what platform do you run this on?

almostEric commented 5 years ago

Windows 10.

I thought it might be some case sensitivity issue but that doesn't seem to be the case. And bfg aborts no matter what I try (I tried deleting all .cpp files as a test)

cschol commented 5 years ago

I'll try it on Windows later today. I ran on Linux.

cschol commented 5 years ago

I tried it on Windows and I get the same results as on Linux. It removes the dep folder from the history correctly.

Now, I get the results you show above if the command is wrong (e.g. the folder name dep is misspelled) OR the command is run twice on a repository, i.e. the first command removes the folder and the second fails if there is no folder named dep left in the history.

If you start fresh, can you post the exact commands and their output?

almostEric commented 5 years ago

I'll give it a shot...

almostEric commented 5 years ago

don't understand. same result. I tried it in just a basic /tmp directory to see if my directory structure was an issue. I also tried running command prompt as an administrator. no difference

almostEric commented 5 years ago

you got to be kidding me. I took off the quotes around 'dep' and it worked

cschol commented 5 years ago

Hmm...what kind of shell are you using? msys2?

almostEric commented 5 years ago

Hi, Sorry for the delay - was working on a new commit before I did this clean up. I am using just the windows shell. msys2 doesn't seem to know where java is, not familiar enough to get it to import windows environment variables

That said, I run BFG (without quotes) and it works - it says it is removing stuff. I clone the repository to frozenwasteland_clean. When I try to push it does nothing and says everything is up to date. I also notice that my branches are gone

cschol commented 5 years ago

Are you trying to push from the locally cloned repo? You need to push from the repo that you ran BFG on, i.e. the bare one. The locally cloned repo will of course be up to date with its origin, which is the bare repo.

almostEric commented 5 years ago

I am following your instructions. It looks like you did a cd back to the original repository and did the push there, so a step was missing. Will try that

almostEric commented 5 years ago

ok. so confused. the original is a mirror which can't be pushed

cschol commented 5 years ago

Hmm, I never had to do this myself (push), but from the BFG documentation:

Finally, once you're happy with the updated state of your repo, push it back up (note that because your clone command used the --mirror flag, this push will update all refs on your remote server):

$ git push

At this point, you're ready for everyone to ditch their old copies of the repo and do fresh clones of the nice, new pristine data. It's best to delete all old clones, as they'll have dirty history that you don't want to risk pushing back into your newly cleaned repo.
almostEric commented 5 years ago

It was just "git push" git push origin master --force didn't work

almostEric commented 5 years ago

shit. That did something but failed

almostEric commented 5 years ago

I looked up error - apparently BFG does not play very well with GitHub. :/

How much of a deal breaker is it that my repository is a little big? The offending directory is long gone..

cschol commented 5 years ago

I cloned master and the dep references are gone. Looks correct to me. The repo is still 41 MB cloned, but you have that eurorack repo in there, which I think is fine. The other option would be a submodule, but the size would be the same either way.

I think this looks good unless Andrew has any objections.

cschol commented 5 years ago

Can you post the error you got?

almostEric commented 5 years ago

This is the article I found with same error as I am getting: https://github.com/rtyley/bfg-repo-cleaner/issues/36

cschol commented 5 years ago

Uh, that looks messy. I would say we leave it as is. I can't see any more obvious offenders that contribute to the size.

Size cloned is now 41MB. About half of what it used to be.

@AndrewBelt?

almostEric commented 5 years ago

bfg looks like it causes more issues than fixes