search

whiskyechobravo / kerkoapp

A web application that provides a faceted search interface for bibliographies managed with Zotero.
https://whiskyechobravo.github.io/kerko/
BSD 3-Clause "New" or "Revised" License
68 stars 24 forks source link

Docker #4

Closed retorquere closed 5 years ago

retorquere commented 5 years ago

For Dockerization, I need kerkoapp to listen on 0.0.0.0 (all interfaces) rather than localhost. Is this possible by config/changing something?

retorquere commented 5 years ago

Never mind -- I have docker working, but when I launch kerkoapp, I get

[2019-07-19 21:22:52,641] ERROR in app: Exception on /bibliography/ [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 36, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.7/site-packages/kerko/views.py", line 39, in search
    criteria, return_fields
  File "/usr/local/lib/python3.7/site-packages/kerko/query.py", line 147, in run_query
    index = open_index()
  File "/usr/local/lib/python3.7/site-packages/kerko/index.py", line 22, in open_index
    raise FileNotFoundError
FileNotFoundError

I've set the .env vars (otherwise kerkoapp would error out earlier because it cannot find the secrets)

current progress visible here.

retorquere commented 5 years ago

I had forgotten to index first. Indexing now but it shows a lot of these:

Arguments: ()
[2019-07-19 21:43:39,182] WARNING in extractors: CreatorTypesExtractor: Missing creator types (I35UAA4B)
--- Logging error ---
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/handlers.py", line 933, in emit
    self.socket.send(msg)
OSError: [Errno 9] Bad file descriptor

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/handlers.py", line 854, in _connect_unixsocket
    self.socket.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/logging/handlers.py", line 936, in emit
    self._connect_unixsocket(self.address)
  File "/usr/local/lib/python3.7/logging/handlers.py", line 865, in _connect_unixsocket
    self.socket.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
Call stack:
  File "/usr/local/bin/flask", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 906, in main
    cli.main(args=args, prog_name=name)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 569, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/flask/cli.py", line 419, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/kerko/cli.py", line 19, in index
    update_index()
  File "/usr/local/lib/python3.7/site-packages/kerko/index.py", line 67, in update_index
    spec.extract_to_document(document, item_context, library_context)
  File "/usr/local/lib/python3.7/site-packages/kerko/specs.py", line 42, in extract_to_document
    return self.extractor.extract(document, self, item_context, library_context)
  File "/usr/local/lib/python3.7/site-packages/kerko/extractors.py", line 141, in extract
    self.warning("Missing creator types", item_context)
  File "/usr/local/lib/python3.7/site-packages/kerko/extractors.py", line 69, in warning
    f"{self.__class__.__name__}: {message} ({item_context.item_key})"
Message: 'CreatorTypesExtractor: Missing creator types (I35UAA4B)'
Arguments: ()
[2019-07-19 21:43:40,238] WARNING in extractors: CreatorTypesExtractor: Missing creator types (AJ22SFDP)
davidlesieur commented 5 years ago

I have tried your fork. After adding the following prestart.sh script:

#!/usr/bin/env sh
flask kerko index

I was able to run the application without issues with the same .env file I am using for the demo site. There might be something in your Zotero library that I have never encountered in my tests. If you don't mind sending me an API key for your library, I could take a look. And if you're interested, I could send you (privately) an API key for you to test with the demo library.

davidlesieur commented 5 years ago

Also, you might want to try setting FLASK_ENV=development in your .env file to get slightly more output messages.

retorquere commented 5 years ago

I did have a prestart file but had forgotten to check it in -- it looks like this (I can't currently push to GH because they have an outage: https://www.githubstatus.com/)

#!/bin/sh

set -x
set -e

flask kerko clean
flask kerko index

With that, and this .env, I get the errors above.

FLASK_APP=kerkoapp.py
FLASK_ENV=production
SECRET_KEY=<REDACTED>
KERKO_TITLE="Kerko Demo"
KERKO_ZOTERO_API_KEY=<REDACTED>
KERKO_ZOTERO_LIBRARY_ID=<REDACTED>
KERKO_ZOTERO_LIBRARY_TYPE=user
KERKO_CSL_STYLE=apa
KERKO_PRINT_ITEM_LINK=True
KERKO_PRINT_CITATIONS_LINK=True
KERKOAPP_EXCLUDE_DEFAULT_FACETS=facet_tag,facet_link
KERKOAPP_COLLECTION_FACETS="KY3BNA6T:110:Topic; 7H2Q7L6I:120:Field of study; JFQRH4X2:130:Type of contribution"
retorquere commented 5 years ago

With development I am seeing a lot of activity:

[2019-07-22 16:10:59,982] DEBUG in index: Document 138 updated (2AQNR8PU): Understanding in Historical Science: Intelligibility and Judgment
[2019-07-22 16:10:59,982] INFO in zotero: Requesting up to 100 child notes for item 3S6EC8H9 starting at position 0...
[2019-07-22 16:11:00,478] DEBUG in index: Document 139 updated (3S6EC8H9): De mechanisering van het wereldbeeld
[2019-07-22 16:11:00,478] INFO in zotero: Requesting up to 100 child notes for item 4BADJRZG starting at position 0...
[2019-07-22 16:11:00,970] DEBUG in index: Document 140 updated (4BADJRZG): Two Gaps in the Contextual Theory of Scientific Understanding
[2019-07-22 16:11:00,970] INFO in zotero: Requesting up to 100 child notes for item 5QXQRP6R starting at position 0...
[2019-07-22 16:11:01,477] DEBUG in index: Document 141 updated (5QXQRP6R): The Contextual Theory of Scientific Understanding: A Rejoinder to Erik Weber
[2019-07-22 16:11:01,477] INFO in zotero: Requesting up to 100 child notes for item 73S9PRBU starting at position 0...
[2019-07-22 16:11:01,997] DEBUG in index: Document 142 updated (73S9PRBU): Thinking About the Ultimate Argument for Realism
[2019-07-22 16:11:01,997] INFO in zotero: Requesting up to 100 child notes for item IZ4ID6EB starting at position 0...
[2019-07-22 16:11:02,508] DEBUG in index: Document 143 updated (IZ4ID6EB): Discussion Note: Making Sense of Understanding
[2019-07-22 16:11:02,510] DEBUG in index: Document 144 updated (4TD6TQXI): Philosophical Papers: Mathematics, matter, and method
[2019-07-22 16:11:02,512] DEBUG in index: Document 145 updated (H7B6SSGR): The Duhem Thesis and the Quine Thesis
[2019-07-22 16:11:02,512] INFO in zotero: Requesting up to 100 child notes for item 2PKNNH6G starting at position 0...
[2019-07-22 16:11:03,005] WARNING in extractors: CreatorTypesExtractor: Missing creator types (2PKNNH6G)
[2019-07-22 16:11:03,009] DEBUG in index: Document 146 updated (2PKNNH6G): In defense of Dogma-notes
[2019-07-22 16:11:03,009] INFO in zotero: Requesting up to 100 child notes for item 37PKUKEZ starting at position 0...
[2019-07-22 16:11:03,516] WARNING in extractors: CreatorTypesExtractor: Missing creator types (37PKUKEZ)
[2019-07-22 16:11:03,519] DEBUG in index: Document 147 updated (37PKUKEZ): Carlyle, sign of times
[2019-07-22 16:11:03,519] INFO in zotero: Requesting up to 100 child notes for item 3GU4Z66Z starting at position 0...
[2019-07-22 16:11:04,017] WARNING in extractors: CreatorTypesExtractor: Missing creator types (3GU4Z66Z)
[2019-07-22 16:11:04,019] DEBUG in index: Document 148 updated (3GU4Z66Z): Descartes
[2019-07-22 16:11:04,019] INFO in zotero: Requesting up to 100 child notes for item 4J2G8BHU starting at position 0...
[2019-07-22 16:11:04,516] WARNING in extractors: CreatorTypesExtractor: Missing creator types (4J2G8BHU)
[2019-07-22 16:11:04,518] DEBUG in index: Document 149 updated (4J2G8BHU): Is There A Human Right To Essential Pharmaceuticals
[2019-07-22 16:11:04,518] INFO in zotero: Requesting up to 100 child notes for item 4VQ3RTBH starting at position 0...
[2019-07-22 16:11:05,011] WARNING in extractors: CreatorTypesExtractor: Missing creator types (4VQ3RTBH)
[2019-07-22 16:11:05,013] DEBUG in index: Document 150 updated (4VQ3RTBH): -Kant's Aesthetics and Teleology (Stanford Encyclopedia of Philosophy)
[2019-07-22 16:11:05,014] INFO in zotero: Requesting up to 100 child notes for item 5RH7PARI starting at position 0...
[2019-07-22 16:11:05,550] WARNING in extractors: CreatorTypesExtractor: Missing creator types (5RH7PARI)
[2019-07-22 16:11:05,553] DEBUG in index: Document 151 updated (5RH7PARI): Strategic Reliabilism, A Naturalistic Approach to Epistemology (J.D. Trout)
[2019-07-22 16:11:05,553] INFO in zotero: Requesting up to 100 child notes for item 63XJJZX5 starting at position 0...
[2019-07-22 16:11:06,041] WARNING in extractors: CreatorTypesExtractor: Missing creator types (63XJJZX5)
[2019-07-22 16:11:06,045] DEBUG in index: Document 152 updated (63XJJZX5): On Truth and Lies in a Nonmoral Sense (1873) (Friedrich Nietzsche)
[2019-07-22 16:11:06,045] INFO in zotero: Requesting up to 100 child notes for item 67J2AHTS starting at position 0...
[2019-07-22 16:11:06,536] WARNING in extractors: CreatorTypesExtractor: Missing creator types (67J2AHTS)
[2019-07-22 16:11:06,541] DEBUG in index: Document 153 updated (67J2AHTS): Moral Epistemology (Stanford Encyclopedia of Philosophy)
davidlesieur commented 5 years ago

Oh, you might want to remove the KERKOAPP_COLLECTION_FACETS line from the .env file. The ids there are specific to the demo library.

retorquere commented 5 years ago

Can I update the index of a running app? If I do the index in prestart it would do it wholly anew every time you restart because the container currently doesn't persist data. What directory should I persist to keep the index across starts?

davidlesieur commented 5 years ago

Yes, you may update the index of a running app. It will keep using the preexisting index until the update is finished. As to where to persist the index, that is a good question. It defaults to a data subdirectory under the app's directory, but maybe we should consider /var/lib/kerkoapp?

retorquere commented 5 years ago

data works for me.

davidlesieur commented 5 years ago

Regarding the warnings "WARNING in extractors: CreatorTypesExtractor: Missing creator types", they appear when indexing items that have no authors, but have no adverse effect on the behavior of the application. I think there is nothing wrong with having no authors, so the warning should probably not be emitted at all.

Regarding the exceptions, they seem to have something to do with the syslog logging handler that's set in kerkoapp.app.config.ProductionConfig.

davidlesieur commented 5 years ago

Yeah, no /dev/log on the container.

retorquere commented 5 years ago

That can be fixed, I can just bind-mount it

retorquere commented 5 years ago

See https://github.com/whiskyechobravo/kerkoapp/pull/6

davidlesieur commented 5 years ago

Fixed by #6