search

NYUCCL / psiTurk

An open platform for science on Amazon Mechanical Turk.
https://psiturk.org
MIT License
277 stars 140 forks source link

python3 macosx Malloc error when entering any shell command? #371

Closed guxuanzhen16 closed 5 years ago

guxuanzhen16 commented 5 years ago

After opening psiturk, server on yields the following error:

[psiTurk server:off mode:sdbx #HITs:0]$ server on
Python(80217,0x1108bc5c0) malloc: *** error for object 0x109f05040: pointer being freed was not allocated
Python(80217,0x1108bc5c0) malloc: *** set a breakpoint in malloc_error_break to debug
Abort trap: 6

I've tried using virtualenv to run psiturk in both Python 2.7.16 and 3.7.4 but the error is the same. Thanks in advance.

xf15 commented 5 years ago

image having the same issue on my brand new mac (unboxed yesterday). psiturk 2.3.3; python 3.7; pip installed through anaconda 3; mac 10.14.6 Help please

xf15 commented 5 years ago

this error disappeared for me when I switched to a conda env in python2.7

guxuanzhen16 commented 5 years ago

In Conda psiturk would still try to run on python3 for some reason, so I had to manual edit each file to run python2.7, which solved the problem for me. It seems psiturk is still not working very steadily on python3.

deargle commented 5 years ago

It works fine on python3 outside of conda -- e.g., in a virtualenv. Passes tests on travis. I don't support conda.

On Wed, Aug 7, 2019, 11:57 AM 玄镇 notifications@github.com wrote:

Closed #371 https://github.com/NYUCCL/psiTurk/issues/371.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7NBEGOEARCQ5E2HDPDQDMEIZA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOS5ZXXTI#event-2540927949, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7PJHX4XNWFNXAC534DQDMEIZANCNFSM4IJMJRRQ .

tchamberlain commented 5 years ago

I was also getting this error using python3 in a virtualenv, i switched to using python 2.7 and the error went away

deargle commented 5 years ago

Is the common thread that everyone here is using conda?

On Tue, Sep 17, 2019, 3:21 PM Taylor Chamberlain notifications@github.com wrote:

I was also getting this error using python3 in a virtualenv, i switched to using python 2.7 and the error went away

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7LVHZOWBNZSENZIG43QKFC6JA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD656HGA#issuecomment-532407192, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7KMNBZRBLSTCCNPSWLQKFC6JANCNFSM4IJMJRRQ .

guxuanzhen16 commented 5 years ago

Before I tried conda I just directly used python3, which did not work either. It seems that it worked for everyone after switching back to 2.7

deargle commented 5 years ago

Right, but I want to solve the python3 part, since python2 is EOL end of this year.

On Wed, Sep 18, 2019, 4:37 PM 玄镇 notifications@github.com wrote:

Before I tried conda I just directly used python3, which did not work either. It seems that it worked for everyone after switching back to 2.7

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7JOZGSBWAFQMNTRI2TQKKUT7A5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7BU62Q#issuecomment-532893546, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7M4YVKJ3JCFVLQGMZLQKKUT7ANCNFSM4IJMJRRQ .

c-j-bates commented 5 years ago

I'm also having the same issue on python 3, but have had success with python 2.7. Do you have any insight into why conda 3 creates a problem here (compared to vanilla venv)?

deargle commented 5 years ago

I can't replicate the error, so I don't have insight. Can you (or anyone experiencing the error) give me as many gory details of your setup as possible? I don't know exactly what to ask for, but maybe what python version and distribution you have installed and how you installed psiturk. Again, psiturk works fine with python 2.7, python 3.6 and python 3.7 on travis, and travis uses a clean environment. See e.g. https://travis-ci.org/NYUCCL/psiTurk/builds/589276088

@Xinzhu-Fang up above mentioned that they're using anaconda 3 on a mac. I'd like to maybe see:

I'm guessing wildly that it's something to do with anaconda-python being used,.

c-j-bates commented 5 years ago

Currently checking with a clean (non-conda) version. I'll let you know.

deargle commented 5 years ago

fyi, travis tests against the following python versions:

c-j-bates commented 5 years ago

I should point out that Xinzhu-Fang and I are both working on OSX. The OP might also be, though they didn't mention. I assume you're using linux, since the travis docs said it is not supported for mac.

deargle commented 5 years ago

Yes, I do all development on a linux (mint) virtual machine. I download my python versions from pyenv.

deargle commented 5 years ago

I just installed anaconda3 into my linux distro and tested psiturk v2.3.5 using anaconda's python3.7.3, which ran without errors.

deargle commented 5 years ago

@Rick-C-137 where were you reading about travis not supporting mac? https://docs.travis-ci.com/user/reference/osx/

c-j-bates commented 5 years ago

I have no trouble on my ubuntu 18 server either with conda. But my laptop is mac, so I want to be able to debug on it. I haven't had success yet in testing if the issue on OSX derives from anaconda, because I'm running into dependency issues with system python installs, which I haven't been able to resolve yet (something to do with not being able to find/use gnureadline).

The red box here: https://docs.travis-ci.com/user/languages/python/ I don't really know anything about travis, so not sure if I'm understanding correctly...

deargle commented 5 years ago

I successfully set up travis to test against osx 10.14.4. No errors from the unit tests. Which leads me to suspect gnureadline. The tests give a compiler warning about it here: https://travis-ci.org/NYUCCL/psiTurk/jobs/590147471#L300 And, the unit tests aren't testing readline input.

Edit: and, cmd2 warns about gnureadline on mac, https://cmd2.readthedocs.io/en/0.9.2/

deargle commented 5 years ago

I noticed that gnureadline was pinned to 6.3.3 in setup.py -- v8.0.0 is now available. But, perhaps it should be up to the user if they want to use gnureadline over whatever default readline version python distro includes. See the big "First... STOP" info at the top of the gnureadline package page. https://pypi.org/project/gnureadline/

I'm betting (hoping) that gnureadline==8.0.0 will fix macosx user woes. But can one (all?) of you try installing psiturk from the head of the python2 branch, using python 3, and tell me if the shell works now? e.g.,

mkvirtualenv -p <path to some version of python3> psiturk-gnureadline-python3-test
git clone git@github.com:NYUCCL/psiTurk.git
cd psiTurk
git checkout python2
pip install -e .

Then, navigate to your psiturk project dir and try launching the shell, and then try running any command. I noticed up above that server on and version both threw the error, so probably any command will throw the error.

Also, @fredcallaway you mentioned something about gnureadline incompatibilities, maybe this will fix things for you, too.

fredcallaway commented 5 years ago

Following these steps I get an error: ModuleNotFoundError: No module named 'psycopg2'

After pip installing, I can start up the shell and post a sandbox hit. No issues with readline!

deargle commented 5 years ago

Great! Would be good if others on the thread can chip in too.

Psycopg2 is required if using a psql db connection, but psiturk lets users use whatever db they want (mysql, psql). Maybe i'll catch and print out an error if psql is in url but if psycopg2 not found.

On Thu, Sep 26, 2019, 5:28 PM Fred Callaway notifications@github.com wrote:

Following these steps I get an error: ModuleNotFoundError: No module named 'psycopg2'

After pip installing, I can start up the shell and post a sandbox hit. No issues with readline!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7IAJIY5SI2VFBVZEMLQLVATDA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7XHTXQ#issuecomment-535722462, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7OMIUD5BM6ZEAP6AN3QLVATDANCNFSM4IJMJRRQ .

c-j-bates commented 5 years ago

I followed your steps above, and got gunicorn errors with the python2 branch. Then I switched back to master and everything worked perfectly. I'm working on homebrew, which I finally got working properly again, so it seems that master works on homebrew python3 but not anaconda. Thanks for your help. Any guesses as to what kind of thing would be different in anaconda vs. homebrew?

deargle commented 5 years ago

Can you show the gunicorn errors when using python 3 (both anaconda and homebrew) on the python2 branch? Should be in server.log. the python2 branch supports both python 3 and python 2, while master is python3 only. So master branch has some compatibility code stripped out, and uses a more recent cmd2 library which is python3 only, which might be fixing whatever was breaking for old gnureadline versions in this thread. Python2 branch is basically stuck on using legacy cmd2 code, which is what is used for the shell.

Be aware that master branch has default enabled a webapi that can be used to post hits, approve and bonus workers, etc. Won't accept default username password from config.txt though.

I didnt do anything different with the master branch yesterday regarding gnureadline. But if you look at that pip gnureadline page, it says that different python distributions (e.g. homebrew, anaconda) come with different static linked versions of readline, which is compiled c, which is what was throwing the malloc errors I think.

On Fri, Sep 27, 2019, 8:46 AM Chris Bates notifications@github.com wrote:

I followed your steps above, and got gunicorn errors with the python2 branch. Then I switched back to master and everything worked perfectly. I'm working on homebrew, which I finally got working properly again, so it seems that master works on homebrew python3 but not anaconda. Thanks for your help. Any guesses as to what kind of thing would be different in anaconda vs. homebrew?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7OPVSZWQRSEIHAXRLTQLYMCNA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ZEG3Y#issuecomment-535970671, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7PXWSYM7HJQADL5BN3QLYMCNANCNFSM4IJMJRRQ .

c-j-bates commented 5 years ago

I'm see this error in the log: ImportError: cannot import name 'InvalidUsageError' from 'psiturk.experiment_errors'

Not sure what it's about...I can give you the whole printout if you'd like.

deargle commented 5 years ago

Ah, yes. Rerun psiturk-setup-example if you are using psiturk from master branch. Some class names changed in custom.py from the python2 branch. Not related to the gnureadline malloc error.

But if you did not get a malloc error when trying to run a command such as version from a psiturk python2-branch shell using python 3, i'll close this as resolved.

On Fri, Sep 27, 2019, 12:14 PM Chris Bates notifications@github.com wrote:

I'm see this error in the log: ImportError: cannot import name 'InvalidUsageError' from 'psiturk.experiment_errors'

Not sure what it's about...I can give you the whole printout if you'd like.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7LF26ARXTQEGNRPP3DQLZEQZA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ZWKLQ#issuecomment-536044846, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7LAJBEH6PGM6VLYXODQLZEQZANCNFSM4IJMJRRQ .

c-j-bates commented 5 years ago

The version command works, but server on fails.

deargle commented 5 years ago

What! Fails with the malloc error specifically, even after using a fresh project dir from a psiturk-setup-example command?

On Fri, Sep 27, 2019, 12:59 PM Chris Bates notifications@github.com wrote:

The version command works, but server on fails.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/NYUCCL/psiTurk/issues/371?email_source=notifications&email_token=AAI6Y7KMIBM3YQBPI2RPUNTQLZJ2NA5CNFSM4IJMJRR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7Z2A2Q#issuecomment-536060010, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI6Y7PK46OL4AR7FPQXPE3QLZJ2NANCNFSM4IJMJRRQ .

c-j-bates commented 5 years ago

No, sorry for being unclear. It was that gunicorn thing (maybe because I didn't remake the example experiment).

deargle commented 5 years ago

Yes, absolutely because you didn't remake the example experiment / because your custom.py generated under psiturk v2 is incompatible with psiturk v3. Thanks for reporting back about malloc being gone! I'm closing this.

TLDR to this whole thread -- psiturk was installing a very old gnureadline that doesn't play nice with python3 on macosx when combined with the last version of the cmd2 library that still supports python2. Head of python2 branch no longer forces install of gnureadline for macosx. If you need it, install it from pip yourself. But most mac osx python distros don't need it.