Pkg data

[ocefpaf]

Not ready for merging yet! We need to discussion how the templatesAndSettings are used to see if pulling those inside the module makes sense or not.

[ocefpaf]

@miguelcleon and @lsetiawan I am not familiar with ODM2-Admin details to know how the templatesAndSettings is used. If the data in templatesAndSettings is needed by the package then moving it into the module is a good idea, but that may require further changes so the module can find it there.

In this PR I only added the necessary change so the templatesAndSettings data is packaged in the tarball.

[miguelcleon]

hi @ocefpaf I ran setup.py in the way you suggested which created the tarball. I unpacked the tarball which created a dist folder. I think your saying templatesAndSettings would also be created in the dist folder, but it was not. I think it would be good to do some reorganization of the project but recommended layouts don't have the templates within the app (http://stackoverflow.com/questions/22841764/best-practice-for-django-project-working-directory-structure). In this example in fact templates and static would be in the same folder as odm2admin. The templates are used to dynamically generate html.

[ocefpaf]

I think your saying templatesAndSettings would also be created in the dist folder, but it was not.

My local tests to show it. Did you tried that in this branch?

I think it would be good to do some reorganization of the project but recommended layouts don't have the templates within the app.

I need to read that as I am unfamiliar with what django recommends. (From strictly packing mindset a package without the needed pkg_data is a broken package.)

I'll read more about this and get back to you.

[miguelcleon]

Ok yes, I had tried from your master branch not pkg_data. Running setup from that branch, unpacking the tar ball did create the templatesAndSettings in the odm2admin folder.

[ocefpaf]

it is working on Travis-CI too, see https://travis-ci.org/miguelcleon/ODM2-Admin/jobs/193404282#L421 all the way to https://travis-ci.org/miguelcleon/ODM2-Admin/jobs/193404282#L463

Running setup from that branch, unpacking the tar ball did create the templatesAndSettings in the odm2admin folder.

You may be using a cached sdist. Maybe removing the folder dist folder and creating it again will show the proper files. We know it is working by looking at Travis-CI logs.

[miguelcleon]

Hi @ocefpaf and @lsetiawan I noticed some problems last Friday and encountered some more today. I wasn't sure if the problem was related to the conda python environment or the changes to the settings file using the new settings.ymal file. At first I thought it was something with the settings changes but eventually a realized it couldn't be and I reverted to the old python environment and that resolved the issues.

problems I encountered were: 1) inline forms were not rendering, I'm guessing this has to do with this import not working properly https://github.com/miguelcleon/ODM2-Admin/blob/master/odm2admin/forms.py#L2

2) ajax lookup fields sometimes weren't working. I haven't been able to figure out why some worked and others did not.

I'd try to provide more info but it's quite late and I need to get off.

[ocefpaf]

hi @ocefpaf I ran setup.py in the way you suggested which created the tarball. I unpacked the tarball which created a dist folder. I think your saying templatesAndSettings would also be created in the dist folder, but it was not.

Can you please try again? We know that it is working for me locally and at Travis-CI but I did add include_package_data=True, which should not be needed, to ensure they are included.

I think it would be good to do some reorganization of the project but recommended layouts don't have the templates within the app (http://stackoverflow.com/questions/22841764/best-practice-for-django-project-working-directory-structure). In this example in fact templates and static would be in the same folder as odm2admin. The templates are used to dynamically generate html.

I find that very confusing and django's own docs recommend otherwise for re-usable apps. See https://docs.djangoproject.com/en/1.10/intro/reusable-apps/#your-project-and-your-reusable-app

Those instructions you sent above do mention a stand-alone website instead of a re-usable app, which gives me the impression that setup.py they have there is only a dummy file with metadata and not a working packaging script.

I could not find an example of a re-usable app with templates are external modules that are shipping together. Although you can create a template separate package to ensure that there won't be namespaces classes. Then templatesAndSettings would be a dependency for odm2admin and users installing odm2admin would not be surprised by the extra folder templatesAndSettings in the site-packages.

My latest commit, https://github.com/miguelcleon/ODM2-Admin/pull/96/commits/d5ddac64050f856fabb357a3b86da133691d0c47, should implement the proper path modification, but I am still unable to run the app. When I type python manage.py runserver I get:

Performing system checks...

System check identified no issues (0 silenced).
Unhandled exception in thread started by <function wrapper at 0x7f9bbf9df230>
Traceback (most recent call last):
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/utils/autoreload.py", line 226, in wrapper
    fn(*args, **kwargs)
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/core/management/commands/runserver.py", line 124, in inner_run
    self.check_migrations()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/core/management/base.py", line 437, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/migrations/executor.py", line 20, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/migrations/loader.py", line 52, in __init__
    self.build_graph()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/migrations/loader.py", line 203, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 65, in applied_migrations
    self.ensure_schema()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/migrations/recorder.py", line 52, in ensure_schema
    if self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor()):
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/base/base.py", line 231, in cursor
    cursor = self.make_debug_cursor(self._cursor())
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/base/base.py", line 204, in _cursor
    self.ensure_connection()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
    self.connect()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/base/base.py", line 199, in ensure_connection
    self.connect()
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/base/base.py", line 171, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/django/db/backends/postgresql/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/home/filipe/miniconda3/envs/ODM2/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
django.db.utils.OperationalError: could not connect to server: Connection refused
        Is the server running on host "127.0.0.1" and accepting
        TCP/IP connections on port 5432?

[miguelcleon]

I'm pretty certain the error is due to the requirement that an instance of an ODM2 postgreSQL database needs to be running. By default in the settings.yaml file that database would be named 'db_name' although we might want to change that. Additionally the command: python manage.py migrate will need to be run before python manage.py runserver The migrate command will create needed django tables in the database for things like user management.

This file can be used to generate an ODM2 schema, within an existing postgreSQL database: https://github.com/ODM2/ODM2/blob/master/src/blank_schema_scripts/postgresql/ODM2_for_PostgreSQL.sql

This file needs to be run to add the additional odm2extra schema, which adds a few extra tables and views and should be run within the same postgreSQL database the ODM2 schema is created in https://github.com/miguelcleon/ODM2-Admin/blob/master/extrasql.sql

[ocefpaf]

I'm pretty certain the error is due to the requirement that an instance of an ODM2 postgreSQL database needs to be running.

The errors do seem to indicate the lack of a database to connect.

By default in the settings.yaml file that database would be named 'db_name' although we might want to change that.

I am looking at settings.yaml and there are many hard-coded information there that may be complicated to deploy in a re-usable App. Maybe that should not be part of the App at all and we could write some doc pages on how to write that file and start the database before running the App.

I see similar apps where devs chose to not be ship as python packages because of that. Instead they ship them in docker images that already have the app and the data configured.

Additionally the command: python manage.py migrate will need to be run before python manage.py runserver The migrate command will create needed django tables in the database for things like user management.

I will try that. Just not sure yet how to start the ODM2 postgreSQL instance.

This file can be used to generate an ODM2 schema, within an existing postgreSQL database: https://github.com/ODM2/ODM2/blob/master/src/blank_schema_scripts/postgresql/ODM2_for_PostgreSQL.sql This file needs to be run to add the additional odm2extra schema, which adds a few extra tables and views and should be run within the same postgreSQL database the ODM2 schema is created in https://github.com/miguelcleon/ODM2-Admin/blob/master/extrasql.sql

This step is similar to the settings.yaml problem I mention above for packaging. I am looking if there are other alternatives or if those steps should be left for the end user to execute.

[miguelcleon]

Yes, a docker image maybe a more appropriate solution.

[lsetiawan]

@ocefpaf @miguelcleon I've started creating a Dockerfile for ODM2-Admin, should I go ahead and try to compose a docker image?

[miguelcleon]

@lsetiawan sounds good to me.

[ocefpaf]

@ocefpaf @miguelcleon I've started creating a Dockerfile for ODM2-Admin, should I go ahead and try to compose a docker image?

I guess that a Dockerfile is the only way to satisfy what is expected for odm2adim. If packaging it, to make a re-usable app, is desired we can re-visit this PR. Closing now because it is not worth fixing the conflicts if we are not moving this forward.

@lsetiawan let me know if you need help if the Dockerfile and how to set hooks with dockerhub to automate the builds.

[miguelcleon]

@ocefpaf @lsetiawan agreed, a Dockerfile seems like the way to go.