search

logv / snorkel

UI for interactive data analysis | https://snorkel.logv.org
https://fb.com/groups/snorkelsnorkelsnorkel
161 stars 21 forks source link

scripts/add_superuser.sh fails #51

Closed niteria closed 4 years ago

niteria commented 4 years ago

Steps:

  1. clone the repository
  2. make setup
  3. make run and Ctrl-C # otherwise I get a secondary error about missing db in the next step, see below
  4. bash scripts/add_superuser.sh niteria

I get:

$ bash scripts/add_superuser.sh niteria
RUNNING COMMAND which sybil
Password: 
Traceback (most recent call last):
  File "/usr/local/bin/flask", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 967, in main
    cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/niteria/tmp/snorkel/src/cli.py", line 75, in add_superuser
    _add_superuser(name)
  File "/home/niteria/tmp/snorkel/src/cli.py", line 68, in _add_superuser
    user_datastore.create_role('superuser')
TypeError: create_role() takes exactly 1 argument (2 given)

The secondary error:

$ bash scripts/add_superuser.sh niteria                                                                  
RUNNING COMMAND which sybil
Traceback (most recent call last):                                                                       
  File "/usr/local/bin/flask", line 10, in <module>
    sys.exit(main())                                                                                     
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 967, in main
    cli.main(args=sys.argv[1:], prog_name="python -m flask" if as_module else None)
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)                         
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)                                                                                
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs) 
  File "/usr/local/lib/python2.7/dist-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/niteria/tmp/snorkel/src/cli.py", line 75, in add_superuser
    _add_superuser(name)
  File "/home/niteria/tmp/snorkel/src/cli.py", line 48, in _add_superuser
    user = User.get(User.email == name)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 5810, in get
    return sq.get()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 6208, in get
    return clone.execute(database)[0]
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 1698, in inner
    return method(self, database, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 1769, in execute
    return self._execute(database)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 1943, in _execute
    cursor = database.execute(self)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2848, in execute
    return self.execute_sql(sql, params, commit=commit)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2842, in execute_sql
    self.commit()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2627, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2833, in execute_sql
    cursor = self.cursor(commit)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2819, in cursor
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2780, in connect
    self._initialize_connection(self._state.conn)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2627, in __exit__
    reraise(new_type, new_type(*exc_args), traceback)
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2777, in connect
    self._state.set_connection(self._connect())
  File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 3069, in _connect
    **self.connect_params)
peewee.OperationalError: unable to open database file

This sequence of steps lets me log in, but the next page fails:

  1. git clone
  2. make setup
  3. make run && Ctrl-C
  4. bash scripts/add_user.sh niteria
  5. bash scripts/add_superuser.sh niteria
  6. make run
  7. open http://127.0.0.1:2333 and login
  8. I land on http://127.0.0.1:2333/datasets which gives me 500 and the following log:
[2020-04-27 15:33:58,389] ERROR in app: Exception on /datasets [GET]  
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "src/auth.py", line 102, in wrapped_func
    return func(*args, **kwargs)
  File "src/web.py", line 111, in get_datasets
    return DatasetsPage(template="datasets.html").render()
  File "/usr/local/lib/python2.7/dist-packages/pudgy/components/page.py", line 110, in render
    self.__prepare__()
  File "src/pages.py", line 46, in __prepare__
    self.context.tables = filter(lambda t: rbac.check("query", t), bs.list_tables())
  File "src/backend/sybil.py", line 377, in list_tables
    tables = run_query_command(["-tables"])
  File "src/backend/sybil.py", line 78, in run_query_command
    ret = run_command(init_cmd_args)
  File "src/backend/sybil.py", line 88, in run_command
    p = Popen(cmd_args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 13] Permission denied

It may be relevant that I already have snorkel.lite pip package installed globally, but I'm not running snorkel.frontend during these tests.

okayzed commented 4 years ago

Can you try running make virtualenv3 and then source dev3/bin/activate before make setup?

niteria commented 4 years ago

I tried:

git clone https://github.com/logv/snorkel.git
cd snorkel
make virtualenv3
source dev3/bin/activate
make setup

And got:

pip install -r requirements.txt
Collecting addict
  Using cached addict-2.2.1-py3-none-any.whl (3.6 kB)
Collecting bcrypt
  Using cached bcrypt-3.1.7-cp34-abi3-manylinux1_x86_64.whl (56 kB)
Collecting flask
  Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Processing /home/niteria/.cache/pip/wheels/ee/29/ed/d3183e79c776b69e51f30f1a2f4de25eceb3c2e947a7ad029d/Flask_Admin-1.5.6-py3-none-any.whl
Collecting flask_dance
  Using cached Flask_Dance-3.0.0-py2.py3-none-any.whl (50 kB)
Processing /home/niteria/.cache/pip/wheels/00/1f/ba/9c168a4e81d61670d0eca017af4d65d8bbc33fe553226c85f0/flask_peewee-3.0.3-py3-none-any.whl
Collecting flask-security
  Using cached Flask_Security-3.0.0-py2.py3-none-any.whl (68 kB)
Processing /home/niteria/.cache/pip/wheels/d1/4b/26/a03d292d10c4ecd0db407e35f8459a574bf87196b7e4875086/parsedatetime-2.5-py3-none-any.whl
Processing /home/niteria/.cache/pip/wheels/b9/f5/45/d010673d2ff29810676faf4c803159b4489a4d0fe62a79ba33/peewee-3.9.2-cp37-cp37m-linux_x86_64.whl
Processing /home/niteria/.cache/pip/wheels/09/d0/e5/a1454de114eec7a9babd58c2411ef01410ee9e02b10fadc6e1/python_geoip-1.2-py3-none-any.whl
Processing /home/niteria/.cache/pip/wheels/3b/2b/c1/61e962b3a42e78383090659465db091de10ac463ac0dfaa27c/python_geoip_geolite2-2015.303-py3-none-any.whl
Processing /home/niteria/.cache/pip/wheels/8d/77/fd/4b7625f5dcaf1d0078b8605373415de02ecb897bcc9c1e6a72/Preparable-0.1.3-py3-none-any.whl
Processing /home/niteria/.cache/pip/wheels/c6/18/4f/9b954a5e21485aab88fd606f8b295b5a6a8c7d69d4225e1cd2/pudgy-0.2.0-py3-none-any.whl
Collecting werkzeug==0.16.1
  Using cached Werkzeug-0.16.1-py2.py3-none-any.whl (327 kB)
Collecting six>=1.4.1
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting cffi>=1.1
  Using cached cffi-1.14.0-cp37-cp37m-manylinux1_x86_64.whl (400 kB)
Collecting itsdangerous>=0.24
  Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Jinja2>=2.10.1
  Using cached Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting click>=5.1
  Downloading click-7.1.2-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 435 kB/s
Collecting wtforms
  Using cached WTForms-2.3.1-py2.py3-none-any.whl (169 kB)
Collecting oauthlib
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting requests-oauthlib>=1.0.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Processing /home/niteria/.cache/pip/wheels/81/45/c8/e9abb602b8a5fcf79ee5de6d65e57b5c7d47b402839e261d8f/URLObject-2.4.3-py3-none-any.whl
Collecting requests>=2.0
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Processing /home/niteria/.cache/pip/wheels/c6/cf/35/7673bace6bf6ed0fe323af2a4a4900c2dc322fd3ecc0241252/wtf_peewee-3.0.0-py3-none-any.whl
Collecting passlib>=1.7
  Using cached passlib-1.7.2-py2.py3-none-any.whl (507 kB)
Collecting Flask-Login>=0.3.0
  Using cached Flask_Login-0.5.0-py2.py3-none-any.whl (16 kB)
Processing /home/niteria/.cache/pip/wheels/97/4c/ee/b508a02fac172ae275847db50f2e1b002b9b5e851d0cf6749d/Flask_Principal-0.4.0-py3-none-any.whl
Processing /home/niteria/.cache/pip/wheels/c3/46/1d/34849a40eb7e4b84d8c794ef49bce76c7d9e968cf0400f2e5c/Flask_Mail-0.9.1-py3-none-any.whl
Collecting Flask-BabelEx>=0.9.3
  Using cached Flask_BabelEx-0.9.4-py3-none-any.whl (8.7 kB)
Collecting Flask-WTF>=0.13.1
  Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
Collecting diskcache
  Using cached diskcache-4.1.0-py2.py3-none-any.whl (44 kB)
Processing /home/niteria/.cache/pip/wheels/7e/fc/21/ac5286aa035303d281ff1a1c2fe6ce3682d2ee2bdcf7f6d89a/pystache-0.5.4-py3-none-any.whl
Collecting libsass
  Downloading libsass-0.20.0-cp36-abi3-manylinux1_x86_64.whl (9.4 MB)
     |████████████████████████████████| 9.4 MB 1.2 MB/s
Collecting pycparser
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Processing /home/niteria/.cache/pip/wheels/22/f5/18/df711b66eb25b21325c132757d4314db9ac5e8dabeaf196eab/blinker-1.4-py3-none-any.whl
Collecting Babel>=1.0
  Using cached Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Processing /home/niteria/.cache/pip/wheels/75/e9/93/a3352ff3d4ef1d8cdeaf6466214d74cf94717239b7835a9a21/speaklater-1.3-py3-none-any.whl
Collecting pytz>=2015.7
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 1.8 MB/s
Installing collected packages: addict, six, pycparser, cffi, bcrypt, itsdangerous, MarkupSafe, Jinja2, werkzeug, click, flask, wtforms, flask-admin, oauthlib, idna, urllib3, certifi, chardet, requests, requests-oauthlib, urlobject, flask-dance, peewee, wtf-peewee, flask-peewee, passlib, Flask-Login, blinker, Flask-Principal, Flask-Mail, pytz, Babel, speaklater, Flask-BabelEx, Flask-WTF, flask-security, parsedatetime, python-geoip, python-geoip-geolite2, preparable, diskcache, pystache, libsass, pudgy
Successfully installed Babel-2.8.0 Flask-BabelEx-0.9.4 Flask-Login-0.5.0 Flask-Mail-0.9.1 Flask-Principal-0.4.0 Flask-WTF-0.14.3 Jinja2-2.11.2 MarkupSafe-1.1.1 addict-2.2.1 bcrypt-3.1.7 blinker-1.4 certifi-2020.4.5.1 cffi-1.14.0 chardet-3.0.4 click-7.1.2 diskcache-4.1.0 flask-1.1.2 flask-admin-1.5.6 flask-dance-3.0.0 flask-peewee-3.0.3 flask-security-3.0.0 idna-2.9 itsdangerous-1.1.0 libsass-0.20.0 oauthlib-3.1.0 parsedatetime-2.5 passlib-1.7.2 peewee-3.9.2 preparable-0.1.3 pudgy-0.2.0 pycparser-2.20 pystache-0.5.4 python-geoip-1.2 python-geoip-geolite2-2015.303 pytz-2020.1 requests-2.23.0 requests-oauthlib-1.3.0 six-1.14.0 speaklater-1.3 urllib3-1.25.9 urlobject-2.4.3 werkzeug-0.16.1 wtf-peewee-3.0.0 wtforms-2.3.1
RESET=1 python src/models.py
Traceback (most recent call last):
  File "src/models.py", line 5, in <module>
    from flask_security import RoleMixin, UserMixin
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask_security/__init__.py", line 13, in <module>
    from .core import Security, RoleMixin, UserMixin, AnonymousUser, current_user
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask_security/core.py", line 28, in <module>
    from .forms import ChangePasswordForm, ConfirmRegisterForm, \
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask_security/forms.py", line 69, in <module>
    email_validator = Email(message='INVALID_EMAIL_ADDRESS')
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/wtforms/validators.py", line 332, in __init__
    raise Exception("Install 'email_validator' for email validation support.")
Exception: Install 'email_validator' for email validation support.
make: *** [Makefile:13: setup-db] Error 1
okayzed commented 4 years ago

I pushed something that should fix this

niteria commented 4 years ago

Ok, the add_superuser.sh now works in py3.

I now get 500 on /datatsets in py3, similar to the one in the first comment:

[2020-05-23 00:17:07,965] ERROR in app: Exception on /datasets [GET]
Traceback (most recent call last):            
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app                   
    response = self.full_dispatch_request()
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/auth.py", line 102, in wrapped_func
    return func(*args, **kwargs)
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/web.py", line 111, in get_datasets
    return DatasetsPage(template="datasets.html").render()
  File "/home/niteria/tmp/snorkel-fresh/snorkel/dev3/lib/python3.7/site-packages/pudgy/components/page.py", line 110, in render
    self.__prepare__()
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/pages.py", line 46, in __prepare__
    self.context.tables = filter(lambda t: rbac.check("query", t), bs.list_tables())
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/backend/sybil.py", line 377, in list_tables
    tables = run_query_command(["-tables"])
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/backend/sybil.py", line 78, in run_query_command
    ret = run_command(init_cmd_args)
  File "/home/niteria/tmp/snorkel-fresh/snorkel/src/backend/sybil.py", line 88, in run_command
    p = Popen(cmd_args, stdin=PIPE, stdout=PIPE, stderr=PIPE)
  File "/usr/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: ''

But the main issue is fixed.

okayzed commented 4 years ago

Is the sybil binary in your path? If not, you can try running 'make sybil' to build sybil and place it in src/backends/sybil/

niteria commented 4 years ago

make sybil fixed it, thank you!