Add CTFd 2.x as distinct framework

bkimminich commented 5 years ago

[x] Identify and list all breaking changes between CTFd 1.x and 2.x data models (https://github.com/CTFd/CTFd/blob/master/CHANGELOG.md)
[x] Add CTFd 2.x as a separate framework with its own generators (or shared where possible with 1.x)
[ ] Rename CTFd framework option into CTFd 1.x
[ ] Make CTFd 1.x the default selection and also map the old alias CTFd to it (for non-breaking change of existing config files - but with deprecation warning upon parsing)

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/66857060-add-ctfd-2-x-as-distinct-framework?utm_campaign=plugin&utm_content=tracker%2F49131683&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F49131683&utm_medium=issues&utm_source=github).

bkimminich commented 5 years ago

CTFd 2.x exports and imports all tables by default. A possible solution would be to export an "empty" CTFd setup with only the mininum required data (i.e. an admin user, a CTF title etc.) and let the util overwrite the challenges.json, keys.json and hints.json files in the ZIP.

:interrobang: @ColdHeat, this would probably overwrite all data someone might already have put in manually, including all users and admin account etc. Is there any way to prevent this from happening?

ColdHeat commented 5 years ago

I think the first thing to know is that while generated exports have every table, the importer doesn't require all tables to exist.

You could generate an import with only challenges.json, flags.json, hints.json, files.json and other juice shop relevant tables. After importing the tables that weren't inside of the zip file won't be affected.

This would work well for situations where the user is importing from the command line because they could modify or otherwise run the import.py script without triggering a database wiping. However, this isn't the same for importing via the web UI where deleting the database before import is difficult to avoid at the moment. I think the best thing to do in that scenario is to allow the wipe to happen and let the import import all challenges but require the user to re-create the admin account and others.

bkimminich commented 5 years ago

Ok, this is exactly what my tool generates, a ZIP with only those tables. But upon UI import there is an error being thrown: KeyError("There is no item named 'db/alembic_version.json' in the archive",) - after that happened all subsequent requests for any pages will end in an Internal Server Error. In the server log I saw OperationalError: (sqlite3.OperationalError) no such table: config [SQL: u'SELECT config.id AS config_id, config."key" AS config_key, config.value AS config_value \nFROM config \nWHERE config."key" = ?\n LIMIT ? OFFSET ?'] [parameters: ('ctf_name', 1, 0)] (Background on this error at: http://sqlalche.me/e/e3q8) then.

So, do I have to provide all other .json files as empty to make the new importer happy?

bkimminich commented 5 years ago

OWASP_Juice_Shop.2018-11-29.CTFd.zip

This is a sample export file my tool generates at the moment, which works up to CTFd 1.2.0 just fine.

ColdHeat commented 5 years ago

You need to migrate the tables that are output because there are significant changes in database structure between 1.2.0 and 2.0.0.

You should look at the 1.2.0 -> 2.0.0 migration script: https://github.com/CTFd/CTFd/blob/master/migrations/1_2_0_upgrade_2_0_0.py

And also the new CTFd models: https://github.com/CTFd/CTFd/blob/master/CTFd/models/__init__.py

Also you do now need to provide db/alembic_version.json but you should be okay to use the alembic_version that's currently in use (8369118943a1) because we should now be able to accept imports from previous version if we do upgrade.

You could try importing your 1.2.0 export into a 1.2.0 CTFd, then using python migrations/1_2_0_upgrade_2_0_0.py, and then exporting to see the difference in what is generated.

mmmod commented 5 years ago

A workaround at this point is to run 1.2.0 of CTFd, import seems to work:

Checkout 1.2.0 of CTFd from the repo
Change db image in docker-compose to "mariadb:10.4" (from mariadb:10.2)

bkimminich commented 5 years ago

Official supported CTFd versions are 1.1.x and 1.2.x at the moment, and they should work out of the box. At least with the singular ctfd/ctfd Docker image.

Am 11. Januar 2019 22:30:01 MEZ schrieb "Markus Örebrand" notifications@github.com:

A workaround at this point is to run 1.2.0 of CTFd, import seems to work:

Checkout 1.2.0 of CTFd from the repo

Change db image in docker-compose to "mariadb:10.4" (from mariadb:10.2)

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/bkimminich/juice-shop-ctf/issues/57#issuecomment-453662790

ColdHeat commented 5 years ago

@bkimminich I think you might have removed your previous comment but I don't believe getting all of the setup data from the user at creation time is necessary.

I think I would just output a the zip with the json files that you want. The others should be ignored by the importer if they're missing.

Also the import_ctf() function has a parameter to control whether or not we erase. Erasing is safer overall but we could modify the command line importer to make erasing optional if it's helpful.

https://github.com/CTFd/CTFd/blob/ae8ce0b4309fa62d5781b26754c6f7463c32e584/CTFd/utils/exports/__init__.py#L103

bkimminich commented 5 years ago

@ColdHeat, thanks, I wrote the now-deleted comment before trying out what you suggested, and that did it. Except for the fact that after import an Internal Server Error pops up and the CTFd server needs a restart. But then all seems okay once you set up your CTF name and admin account again.

I attached a ZIP for CTFd 2.x, so if you can give me any hint what to add in order to prevent this error, I'll happily change my tool a bit.

OWASP_Juice_Shop.2019-01-29.CTFd2.zip

ColdHeat commented 5 years ago

Are you referring to the following?

Traceback (most recent call last):
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 1741, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/kchung/Repositories/CTFd/CTFd/views.py", line 122, in setup
    return render_template('setup.html', nonce=session.get('nonce'))
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/templating.py", line 134, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/jinja2/environment.py", line 869, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/jinja2/loaders.py", line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/templating.py", line 58, in get_source
    return self._get_source_fast(environment, template)
  File "/Users/kchung/.virtualenvs/ctfd/lib/python2.7/site-packages/flask/templating.py", line 83, in _get_source_fast
    return loader.get_source(environment, template)
  File "/Users/kchung/Repositories/CTFd/CTFd/__init__.py", line 84, in get_source
    template = "/".join([theme, 'templates', template])
TypeError: sequence item 0: expected string, NoneType found

This might be a bug in CTFd ultimately but you can fix it by providing a config.json with ctf_theme set to core.

Importing from the command line worked fine for me.

bkimminich commented 5 years ago

This is my log after the error popup right after imprting the attached ZIP file:

* Loaded module, <module 'CTFd.plugins.challenges' from '/opt/CTFd/CTFd/plugins/challenges/__init__.py'>
 * Loaded module, <module 'CTFd.plugins.dynamic_challenges' from '/opt/CTFd/CTFd/plugins/dynamic_challenges/__init__.py'>
 * Loaded module, <module 'CTFd.plugins.flags' from '/opt/CTFd/CTFd/plugins/flags/__init__.py'>
Starting CTFd
[2019-01-29 20:15:05 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2019-01-29 20:15:05 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
[2019-01-29 20:15:05 +0000] [1] [INFO] Using worker: geventwebsocket.gunicorn.workers.GeventWebSocketWorker
[2019-01-29 20:15:05 +0000] [27] [INFO] Booting worker with pid: 27
 * Loaded module, <module 'CTFd.plugins.challenges' from '/opt/CTFd/CTFd/plugins/challenges/__init__.pyc'>
 * Loaded module, <module 'CTFd.plugins.dynamic_challenges' from '/opt/CTFd/CTFd/plugins/dynamic_challenges/__init__.pyc'>
 * Loaded module, <module 'CTFd.plugins.flags' from '/opt/CTFd/CTFd/plugins/flags/__init__.pyc'>
[2019-01-29 20:15:53,957] ERROR in app: Exception on /setup [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/CTFd/CTFd/views.py", line 122, in setup
    return render_template('setup.html', nonce=session.get('nonce'))
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 134, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 869, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 58, in get_source
    return self._get_source_fast(environment, template)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 83, in _get_source_fast
    return loader.get_source(environment, template)
  File "/opt/CTFd/CTFd/__init__.py", line 84, in get_source
    template = "/".join([theme, 'templates', template])
TypeError: sequence item 0: expected string, NoneType found
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 976, in handle_one_response
    self.run_application()
  File "/usr/local/lib/python2.7/site-packages/geventwebsocket/handler.py", line 87, in run_application
    return super(WebSocketHandler, self).run_application()
  File "/usr/local/lib/python2.7/site-packages/gevent/pywsgi.py", line 923, in run_application
    self.result = self.application(self.environ, self.start_response)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/flask_socketio/__init__.py", line 43, in __call__
    start_response)
  File "/usr/local/lib/python2.7/site-packages/engineio/middleware.py", line 67, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 2295, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/local/lib/python2.7/site-packages/flask_restplus/api.py", line 583, in error_router
    return original_handler(e)
  File "/usr/local/lib/python2.7/site-packages/flask/app.py", line 1748, in handle_exception
    return self.finalize_request(handler(e), from_error_handler=True)
  File "/opt/CTFd/CTFd/errors.py", line 16, in general_error
    return render_template('errors/500.html'), 500
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 134, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 869, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 58, in get_source
    return self._get_source_fast(environment, template)
  File "/usr/local/lib/python2.7/site-packages/flask/templating.py", line 83, in _get_source_fast
    return loader.get_source(environment, template)
  File "/opt/CTFd/CTFd/__init__.py", line 84, in get_source
    template = "/".join([theme, 'templates', template])
TypeError: sequence item 0: expected string, NoneType found
2019-01-29T20:15:54Z {'REMOTE_PORT': '52784', 'HTTP_HOST': '192.168.99.100:32768', 'REMOTE_ADDR': '192.168.99.1', (hidden keys: 25)} failed with TypeError

OWASP_Juice_Shop.2019-01-29.CTFd2.zip

bkimminich commented 5 years ago

The error is not a big deal, though. I wrote a note in the output of my tool:

lock[bot] commented 4 years ago

This thread has been automatically locked because it has not had recent activity after it was closed. :lock: Please open a new issue for regressions or related bugs.

juice-shop / juice-shop-ctf

Add CTFd 2.x as distinct framework #57