test and enable dev installation on mac and windows

[cittadhammo]

Hi,

My install so far

on mac catalina I had to use the following command while following your guide when facing errors https://github.com/suttacentral/bilara/wiki/Dev-Installation:

git clone https://github.com/suttacentral/bilara.git

brew install pyenv-virtualenv

pip freeze > requirements.txt

install pyenv from https://github.com/pyenv/pyenv#homebrew-in-macos with

brew update brew install pyenv

add this to .zshrc

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

then

exec "$SHELL"

typo in ./run-dev (no underscore)

I installed https://docs.docker.com/get-docker/ This part of the guide is a bit confusing. I never used docker before and there is no link for what to do.

open the app and wait for it to start.

docker run -p 8529:8529 -e ARANGO_ROOT_PASSWORD=openSesame arangodb/arangodb:3.9.2

then arangoDB should be running in the docker app.

My error

if using Suttacentral you can start the docker instance What does this mean ?

~ % docker start sc-arangodb
Error response from daemon: No such container: sc-arangodb
Error: failed to start containers: sc-arangodb

I can brower http://localhost:3003/ but then clicking on login it fails to Service Unavailable

[blake-sc]

if using Suttacentral you can start the docker instance What does this mean ?

If you have installed the suttacentral ecosystem from https://github.com/suttacentral/suttacentral it includes a bunch of docker containers, including one for ArangoDB, and it's perfectly fine to have Bilara use the same database as the SuttaCentral app.

If you don't have the SuttaCentral ecosystem installed you can just install ArangoDB in any way you please, docker is a good approach but using brew should be completely fine too. If using docker you should probably give the container a name for ease of issuing commands to it.

[cittadhammo]

OK so I've been running ArangoDB :

Initializing root user...Hang on...
2022-06-28T16:58:29Z [8] INFO [f6e0e] {aql} memory limit per AQL query automatically set to 1249566720 bytes. to modify this value, please adjust the startup option `--query.memory-limit`
2022-06-28T16:58:29Z [8] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2022-06-28T16:58:30Z [8] INFO [c1b63] {arangosearch} ArangoSearch maintenance: [1..1] commit thread(s), [1..1] consolidation thread(s)
2022-06-28T16:58:30Z [8] INFO [7da27] {startup} server will now shut down due to upgrade, database initialization or admin restoration.
Initializing database...Hang on...
Database initialized...Starting System...
2022-06-28T16:58:44Z [1] INFO [e52b0] {general} ArangoDB 3.9.2 [linux] 64bit, using jemalloc, build tags/v3.9.2-0-g8bf70c5f5b6, VPack 0.1.35, RocksDB 6.27.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1o  3 May 2022
2022-06-28T16:58:44Z [1] INFO [75ddc] {general} detected operating system: Linux version 5.10.104-linuxkit (root@buildkitsandbox) (gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, GNU ld (GNU Binutils) 2.35.2) #1 SMP Thu Mar 17 17:08:06 UTC 2022
2022-06-28T16:58:44Z [1] INFO [25362] {memory} Available physical memory: 2082611200 bytes, available cores: 2
2022-06-28T16:58:44Z [1] INFO [3bb7d] {cluster} Starting up with role SINGLE
2022-06-28T16:58:44Z [1] INFO [f6e0e] {aql} memory limit per AQL query automatically set to 1249566720 bytes. to modify this value, please adjust the startup option `--query.memory-limit`
2022-06-28T16:58:44Z [1] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2022-06-28T16:58:45Z [1] INFO [fe333] {engines} RocksDB recovery starting, scanning WAL starting from sequence number 134, latest sequence number: 208
2022-06-28T16:58:45Z [1] INFO [a4ec8] {engines} RocksDB recovery finished, WAL entries scanned: 75, recovery start sequence number: 134, latest WAL sequence number: 208, max tick value found in WAL: 116, last HLC value found in WAL: 0
2022-06-28T16:58:45Z [1] INFO [c1b63] {arangosearch} ArangoSearch maintenance: [1..1] commit thread(s), [1..1] consolidation thread(s)
2022-06-28T16:58:45Z [1] INFO [6ea38] {general} using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2022-06-28T16:58:46Z [1] INFO [cf3f4] {general} ArangoDB (version 3.9.2 [linux]) is ready for business. Have fun!

I've then created the user and database "bilara" everywhere

Then clone the suttacentral/bilara-data in a folder named repo.

then I run run-dev and get:

    up to date, audited 1112 packages in 7s

    60 packages are looking for funding
      run `npm fund` for details

    14 vulnerabilities (8 high, 6 critical)

    To address issues that do not require attention, run:
      npm audit fix

    To address all issues (including breaking changes), run:
      npm audit fix --force

    Run `npm audit` for details.
    ./run-dev: line 5: poetry: command not found
    ./run-dev-fast: line 12: poetry: command not found
    es-dev-server started on http://localhost:3003
      Serving files from '/Users/cittadhammo/github/bilara/client'.
      Using history API fallback, redirecting route requests to '/index.html'
      Using auto compatibility mode, transforming code on older browsers based on user agent.

then open the browser click on login and still Service Unavailable

[blake-sc]

Actually now I look at it, the dev installations wiki page is much more out of date than the page for installation on Ubuntu: https://github.com/suttacentral/bilara/wiki/Dev-Installation-Ubuntu-20.04

If you follow the instructions under poetry and python dependencies (which I think should work on mac) then that should resolve the error.

[cittadhammo]

OK so I installed pyenv install 3.8.2 then poetry install

when I run run-dev, I still cannot login in localhost:3003

up to date, audited 1112 packages in 5s

60 packages are looking for funding
  run `npm fund` for details

14 vulnerabilities (8 high, 6 critical)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
Installing dependencies from lock file

No dependencies to install or update
No new migrations.
es-dev-server started on http://localhost:3003
  Serving files from '/Users/cittadhammo/github/bilara/client'.
  Using history API fallback, redirecting route requests to '/index.html'
  Using auto compatibility mode, transforming code on older browsers based on user agent.
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 299-972-654
127.0.0.1 - - [29/Jun/2022 10:10:50] "GET / HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/cittadhammo/github/bilara/server/app.py", line 19, in <module>
    import git_fs
  File "/Users/cittadhammo/github/bilara/server/git_fs.py", line 16, in <module>
    from git_branch import GitBranch, base_repo
  File "/Users/cittadhammo/github/bilara/server/git_branch.py", line 11, in <module>
    base_repo = Repo(REPO_DIR)
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/git/repo/base.py", line 224, in __init__
    self.working_dir: Optional[PathLike] = self._working_tree_dir or self.common_dir
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/git/repo/base.py", line 307, in common_dir
    raise InvalidGitRepositoryError()
git.exc.InvalidGitRepositoryError
127.0.0.1 - - [29/Jun/2022 10:10:52] "GET /?__debugger__=yes&cmd=resource&f=jquery.js HTTP/1.1" 200 -
127.0.0.1 - - [29/Jun/2022 10:10:52] "GET /?__debugger__=yes&cmd=resource&f=debugger.js HTTP/1.1" 200 -
127.0.0.1 - - [29/Jun/2022 10:10:52] "GET /?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -
127.0.0.1 - - [29/Jun/2022 10:10:52] "GET /?__debugger__=yes&cmd=resource&f=ubuntu.ttf HTTP/1.1" 200 -
127.0.0.1 - - [29/Jun/2022 10:10:53] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -
127.0.0.1 - - [29/Jun/2022 10:10:53] "GET /?__debugger__=yes&cmd=resource&f=console.png HTTP/1.1" 200 -

[blake-sc]

Try setting the file server/local_config.py to this:

config = {
    'GH_REPO': 'suttacentral/bilara-data',
    'GIT_REMOTE_REPO': 'https://github.com/suttacentral/bilara-data.git',
    'GIT_SYNC_ENABLED': False,
}

Also delete the repo folder.

Then with luck run-dev will setup everything correctly.

[cittadhammo]

Hi, this is my config file that i've updated

import logging
from os import environ

import pathlib

class Config(dict):
    def __getattr__(self, key):
        return self[key]

BASE_DIR = pathlib.Path(__file__).absolute().parent.parent

config = Config({

    # If GITHUB_AUTH is enabled then git details should be provided
    'GITHUB_AUTH_ENABLED': False,

    # If disabled, no push/pull is performed.
    'GIT_SYNC_ENABLED': False,

    # If disabled, the repo dir will be treated as a normal directory
    'GIT_COMMIT_ENABLED': False,

    'GIT_APP_KEY': '',
    'GIT_APP_SECRET': '',

    'GIT_USER': '',
    'GIT_PASSWORD': '',

    # Git Repo
    'GH_REPO': 'suttacentral/bilara-data',
    'GIT_REMOTE_REPO': 'https://github.com/suttacentral/bilara-data.git',
    'REPO_DIR':  BASE_DIR / 'repo',
    'CHECKOUTS_DIR': BASE_DIR / 'checkouts',

    'PUBLISHED_BRANCH_NAME': 'published',
    'UNPUBLISHED_BRANCH_NAME': 'unpublished',
    'REVIEW_BRANCH_NAME': 'review',

    'SECRET': 'CHANGE ME',

    'ARANGO_USER': 'root',
    'ARANGO_PASSWORD': 'test',
    'ARANGO_DB_NAME': 'bilara',

    'PUSHOVER_TOKEN': '',
    'PUSHOVER_ADMIN_KEY': '',

    'LOCAL_USERNAME': 'Bob',
    'LOCAL_LOGIN': 'Bob',
    'LOCAL_EMAIL': 'bob@example.com'
})

try:
    import local_config
    config.update(local_config.config)
except ImportError:
    logging.warning('local_config.py does not exist')

GH_REPO = config.GH_REPO
GIT_REMOTE_REPO = config.GIT_REMOTE_REPO
REPO_DIR = config.REPO_DIR
CHECKOUTS_DIR = config.CHECKOUTS_DIR

PUBLISHED_BRANCH_NAME = config.PUBLISHED_BRANCH_NAME
UNPUBLISHED_BRANCH_NAME = config.UNPUBLISHED_BRANCH_NAME
GIT_SYNC_ENABLED = config.GIT_SYNC_ENABLED

WORKING_DIR = CHECKOUTS_DIR / UNPUBLISHED_BRANCH_NAME

TM_ALIAS = config.get('TM_ALIAS', {})

if GIT_SYNC_ENABLED:
    GITHUB_ACCESS_TOKEN = config.GITHUB_ACCESS_TOKEN
else:
    GITHUB_ACCESS_TOKEN = None

When runing run-dev It did make new copy of repo folder, but I still cannot login in and get Service Unavailable

I might give a try on anther ubuntu computer and give up install on mac. Unless you would like me to apply perseverance ;-)

[cittadhammo]

Ho I can see that ArangoDB password is not the good one. I give it another try ;-) OK it doesn't make any difference...

by the way this is how I run ArangoDB docker run -p 8529:8529 -e ARANGO_ROOT_PASSWORD=test arangodb/arangodb:3.9.2

it shoots :

Initializing root user...Hang on...
2022-07-02T07:56:10Z [8] INFO [f6e0e] {aql} memory limit per AQL query automatically set to 1249566720 bytes. to modify this value, please adjust the startup option `--query.memory-limit`
2022-07-02T07:56:10Z [8] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2022-07-02T07:56:12Z [8] INFO [c1b63] {arangosearch} ArangoSearch maintenance: [1..1] commit thread(s), [1..1] consolidation thread(s)
2022-07-02T07:56:12Z [8] INFO [7da27] {startup} server will now shut down due to upgrade, database initialization or admin restoration.
Initializing database...Hang on...
Database initialized...Starting System...
2022-07-02T07:56:23Z [1] INFO [e52b0] {general} ArangoDB 3.9.2 [linux] 64bit, using jemalloc, build tags/v3.9.2-0-g8bf70c5f5b6, VPack 0.1.35, RocksDB 6.27.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1o  3 May 2022
2022-07-02T07:56:23Z [1] INFO [75ddc] {general} detected operating system: Linux version 5.10.104-linuxkit (root@buildkitsandbox) (gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, GNU ld (GNU Binutils) 2.35.2) #1 SMP Thu Mar 17 17:08:06 UTC 2022
2022-07-02T07:56:23Z [1] INFO [25362] {memory} Available physical memory: 2082611200 bytes, available cores: 2
2022-07-02T07:56:23Z [1] INFO [3bb7d] {cluster} Starting up with role SINGLE
2022-07-02T07:56:23Z [1] INFO [f6e0e] {aql} memory limit per AQL query automatically set to 1249566720 bytes. to modify this value, please adjust the startup option `--query.memory-limit`
2022-07-02T07:56:23Z [1] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2022-07-02T07:56:23Z [1] INFO [fe333] {engines} RocksDB recovery starting, scanning WAL starting from sequence number 131, latest sequence number: 205
2022-07-02T07:56:23Z [1] INFO [a4ec8] {engines} RocksDB recovery finished, WAL entries scanned: 75, recovery start sequence number: 131, latest WAL sequence number: 205, max tick value found in WAL: 116, last HLC value found in WAL: 0
2022-07-02T07:56:23Z [1] INFO [c1b63] {arangosearch} ArangoSearch maintenance: [1..1] commit thread(s), [1..1] consolidation thread(s)
2022-07-02T07:56:23Z [1] INFO [6ea38] {general} using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2022-07-02T07:56:24Z [1] INFO [cf3f4] {general} ArangoDB (version 3.9.2 [linux]) is ready for business. Have fun!

And then ./run-dev

up to date, audited 1112 packages in 11s

60 packages are looking for funding
  run `npm fund` for details

14 vulnerabilities (8 high, 6 critical)

To address issues that do not require attention, run:
  npm audit fix

To address all issues (including breaking changes), run:
  npm audit fix --force

Run `npm audit` for details.
Installing dependencies from lock file

No dependencies to install or update
Migration complete: 1-initialize

Create `strings` collection.
Create `ngrams5` analyzer.
Create `strings_ngram_view` view.

Migration complete: 2-add-splitter
Migration complete: 3-add-simple-normalizer
Migration complete: 4-add-paths
Migration complete: 5-add-historic
Migrations completed: 5
es-dev-server started on http://localhost:3003
  Serving files from '/Users/cittadhammo/github/bilara/client'.
  Using history API fallback, redirecting route requests to '/index.html'
  Using auto compatibility mode, transforming code on older browsers based on user agent.
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 299-972-654
Pulling Repo (one time only)

this is in the localhost 5000

and certainly my checkout folder is empty...

[cittadhammo]

'LOCAL_USERNAME': 'Bob',
    'LOCAL_LOGIN': 'Bob',
    'LOCAL_EMAIL': 'bob@example.com'

is this normal or should it be bilara bilara ?

[blake-sc]

That's pretty close to working. I pushed an update which should fix the FileNotFoundError with new installs.

The "LOCAL_" stuff is for masquerading as a Bilara user.

[cittadhammo]

OK So I've started a new install and follow the previous steps. Now I get :

flask.cli.NoAppException: While importing "app", an ImportError was raised:

Traceback (most recent call last):
File "/Users/cittadhammo/.pyenv/versions/3.7.2/envs/bilara/lib/python3.7/site-packages/flask/cli.py", line 240, in locate_app
__import__(module_name)
File "/Users/cittadhammo/github/bilara/server/app.py", line 19, in <module>
import git_fs
File "/Users/cittadhammo/github/bilara/server/git_fs.py", line 4, in <module>
from git import Repo, GitCommandError
File "/Users/cittadhammo/.pyenv/versions/3.7.2/envs/bilara/lib/python3.7/site-packages/git/__init__.py", line 8, in <module>
from git.exc import * # @NoMove @IgnorePep8
File "/Users/cittadhammo/.pyenv/versions/3.7.2/envs/bilara/lib/python3.7/site-packages/git/exc.py", line 15, in <module>
from git.types import PathLike
File "/Users/cittadhammo/.pyenv/versions/3.7.2/envs/bilara/lib/python3.7/site-packages/git/types.py", line 13, in <module>
from typing_extensions import (Final, Literal, SupportsIndex, # noqa: F401
ModuleNotFoundError: No module named 'typing_extensions'

[blake-sc]

Hmm, that's a new one. I think it's because of the version of python, that stack trace is from python 3.7 while it should be running under python 3.8 or higher.

Previously in your stacktrace it was correctly running in the poetry virtualenv python 3.8

File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise

[cittadhammo]

yes I tried at some point to activate the correct version with pyenv because I got errors, but I may have missed something. I think I should have started with a new terminal when I did my new install. I'll give it another try... because my python version is 3.8 on my machine

I'll try again later ;-)

[cittadhammo]

Ok, so my error came from the instruction in https://github.com/suttacentral/bilara/wiki/Dev-Installation

pyenv install 3.7.2
cd ~/bilara/server
pyenv virtualenv 3.7.2 bilara
pyenv local bilara
pip install -r requirements.txt

So I've done in with 3.8.2 now and I get, when running run-dev a permission error

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/cittadhammo/github/bilara/server/arango_common.py", line 16, in run_migrations
    if not db.has_collection('meta'):
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/arango/database.py", line 859, in has_collection
    return any(col['name'] == name for col in self.collections())
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/arango/database.py", line 884, in collections
    return self._execute(request, response_handler)
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/arango/api.py", line 66, in _execute
    return self._executor.execute(request, response_handler)
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/arango/executor.py", line 82, in execute
    return response_handler(resp)
  File "/Users/cittadhammo/Library/Caches/pypoetry/virtualenvs/bilara-server-ZZ45nMd0-py3.8/lib/python3.8/site-packages/arango/database.py", line 875, in response_handler
    raise CollectionListError(resp, request)
arango.exceptions.CollectionListError: [HTTP 401][ERR 11] not authorized to execute this request
es-dev-server started on http://localhost:3003
  Serving files from '/Users/cittadhammo/github/bilara/client'.
  Using history API fallback, redirecting route requests to '/index.html'
  Using auto compatibility mode, transforming code on older browsers based on user agent.

I tried to do run-dev with sudo, but same thing happened

[cittadhammo]

I must have made a mistake while configuring the database. I tried again and It is running:

No dependencies to install or update
Migration complete: 1-initialize

Create `strings` collection.
Create `ngrams5` analyzer.
Create `strings_ngram_view` view.

Migration complete: 2-add-splitter
Migration complete: 3-add-simple-normalizer
Migration complete: 4-add-paths
Migration complete: 5-add-historic
Migrations completed: 5
es-dev-server started on http://localhost:3003
  Serving files from '/Users/cittadhammo/github/bilara/client'.
  Using history API fallback, redirecting route requests to '/index.html'
  Using auto compatibility mode, transforming code on older browsers based on user agent.
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
WARNING:root:local_config.py does not exist
Pulling Repo (one time only)
 * Debugger is active!
 * Debugger PIN: 258-815-931
WARNING:root:local_config.py does not exist

I cannot see the debug at http://127.0.0.1:5000/ page loading, it dosen't show anything

[cittadhammo]

I've added local-config

config = {
    'LOCAL_USERNAME': 'sujato',
    'LOCAL_LOGIN': 'sujato',
    'LOCAL_EMAIL': 'bob@example.com',
    'GIT_SYNC_ENABLED': False,
}

and I still get the same problem, cannot see debuger.

No dependencies to install or update
No new migrations.
es-dev-server started on http://localhost:3003
  Serving files from '/Users/cittadhammo/github/bilara/client'.
  Using history API fallback, redirecting route requests to '/index.html'
  Using auto compatibility mode, transforming code on older browsers based on user agent.
 * Environment: development
 * Debug mode: on
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
Pulling Repo (one time only)
 * Debugger is active!
 * Debugger PIN: 258-815-931
 * ```