An online reference for cryptic crossword solutions with an indication as how the answer can be derived from the clue and linked to the setter with a measure of the difficulty level associated with their clues.
The project is also intended to help better understand the deployment lifecycle using Amazon AWS.
The application is intended to be provisioned as an ElasticBeanstalk instance.
The application works best with Python 3.6 or later.
The application is written using the lightweight Flask framework.
The virtualenv module can help provide a packageable instance of the whole application but is not required when deploying direct to AWS.
Uwsgi provides the middleware layer and provides the integration with an http server such as nginx or Apache httpd; Amazon AWS uses nginx for the frontend web server.
Accessing the sections of the application is achieved through a RESTful interface in the following forms:
The routes can specify the methods over which data can be submitted. index and show routes should only accept GET requests, while new and edit routes can also accept POST requests when new or updated data is submitted via a form.
The front page of the application provides an input box to prompt for a cue word that is often found in clues. a JQuery AJAX request is used to pass at least two letters to a route that searches particular table and presents an autocomplete list from which a matching word can be selected. The request includes a callback reference in the query string which needs to be included in the JSONP response.
The database schema is modelled in the application using the Peewee module.
Peewee is used as the ORM for the database because of its and the database's simplicity.
Despite looking that it should be possible to retrieve columns from multiple tables using inner joins it doesn't actually seem to work.
The following tables are solution_types_edit
Records the inserts, updates and deletions of records from the tables.
Records the name and brief description of a published crossword setter. There is also an indication as to the general skill level required to solve the clues.
Records the details of a crossword clues and solutions along with the type of clue and a hint showing how to get the solution from the clue.
A lookup table of key words in a clue that can indicate the method required to solve part or all of the clue. This will often, but not always, indicate the type of clue; sometimes the cue is just for a few letters of the solution.
Records a simple ranking order for the level of difficulty associated with the setter.
Stores the name a brief description of the type of clue.
If the template directory structure has more than a passing resemblance to Ruby on Rails, it's not a coincidence; perhaps Rails took the ideas from elsewhere, but it works well and encourages consistency.
The display-related templates are kept in the views sub-directory of the templates folder. There is a template directory for each route 'controller' or database model class:
Each of these template directories will typically contain the following Jinja2 templates
There are two general template directories that can be used by any of the main application templates.
The number of entries on the crossword_solutions page mean that some pagination arrangement needs to be deployed to keep the index view usable. Rather than use the Flask-paginate pip (where stylesheet problems were experienced) a simpler method was employed based on a Flask snippet.
This is further documented at my blog.
The method could just as easily be applied to other application sections.
One of the goals for developing this project is to get an insight into maintaining a test regime as features are developed.
The unit tests are written using the Python unittest framework. The emphasis is on checking the database functionality, trying to trigger integrity violations where we'd expect them, and emulating page navigation and form submission, including responses to bad input.
To run the tests use,
APP_SETTINGS='test-settings.py' python3 crossword-hints-tests.pyThere is a pip module, Flask-login, that provides a simple framework for application authentication. Rather than have to maintain a full database of user accounts, passwords and privileges it might be easier to use an LDAP service for the authentication.
Setting up a simple directory service and how to get a Flask application to authenticate users through it is described at:
There are questions as to how this would operate in an AWS environment.
The Flask-login protecting views section states that the login_required decorator applied to restricted routes can be disabled by adding the following settings
LOGIN_DISABLED=TrueThe format of entries for activity logging are:
For operations where there is no specific user 'system' can be used.
There is no difference in setting this up from what would be done on a typical Linux development system. Install Windows Services for Linux an install a Ubuntu (or other) image from the Windows store.
$ sudo apt-get install libsasl2-dev python-dev libldap2-dev libssl-dev python3-pip python-minimal
$ pip3 install virtual env
$ pip3 install -r requirements.txt
$ cd crossword-hints
$ python3 -m virtualenv .
$ source ./bin/activate
$ FLASK_APP=crossword_hints.py flask runThe service will now be accessible on http://127.0.0.1:5000/
As described at https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html?icmpid=docs_elasticbeanstalk_console
$ pip install awsebcli --upgrade --userdrop the '--user' option if installing as root, as will be necessary if the commands are to be run from a Jenkins jobs.
Do not use the root user to deploy the application. Use the AWS IAM dashboard to create suitable groups and users with appropriate policies to enable the application environment to be deployed.
For a simple Flask application, the following policies should suffice
Create a group, say, flask-apps, with the above policies applied and then add a specific application user, say, crossword-hints. Create an access key pair and store them safely and never store them in the appliction files.
$ eb initThe application file needs to define an application context called application for it to be loadable
$ eb create crossword-hints --envvars SECRET_KEY="MySeCrEtKeY"Original development was with an application file called crossword_hints.py until deployment was switched to AWS. After the application environment has been created it may well be necessary to manually edit the configuration and change the WSGIPath from the default value of 'application.py'. See https://stackoverflow.com/questions/31169260/your-wsgipath-refers-to-a-file-that-does-not-exist, https://stackoverflow.com/questions/20558747/how-to-deploy-structured-flask-app-on-aws-elastic-beanstalk and http://blog.uptill3.com/2012/08/25/python-on-elastic-beanstalk.html for more details if not an authoratative answer. Setting the WSGIPath and SECRET_KEY variables on the instances can be done by including a file, say, options.config, in the .ebextensions directory
option_settings:
aws:elasticbeanstalk:application:environment:
SECRET_KEY: ChangeMe
aws:elasticbeanstalk:container:python:
WSGIPath: crossword_hints.pyDespite changing the application filename from the EB default, the application context still needs to be called application (rather than 'app' as used in just about all Flask examples).
Use a multibranch pipeline job to prepare, test and deploy the application in stages.
To build a standalone copy of the application from a git checkout we can use virtualenv and make sure all the required libraries are installed
virtualenv crossword-hints
cd crossword-hints
. ./bin/activate
pip install -r requirements.txtThe file, requirements.txt, is also used by AWS ElasticBeanstalk to identify the application dependencies and install them before starting the environment.
Create an archive of the codebase to run the application
$ git archive -v -o crossword-hints.zip --format=zip HEADIf using the EB CLI there is no need to create the archive, code updates can be applied directly, but this is still useful when deploying via the console.
To try out new settings on a particular environment before committing to the git repo we can deploy just the candidate files (from git add ...):
$ eb deploy --staged crossword-hints-devSources of additional information.