search

cyfronet-fid / eosc-search-service

GNU General Public License v3.0
1 stars 1 forks source link

readme

EOSC Search Service

Table of contents

Running the Backend with a docker

Run docker-compose up --build.

UI

IMPORTANT!!! UI working directory is ui, commands will work only in the catalog.

Compile time environmental variables

UI uses ng-node-environment to generate variables. Please consult the documentation of the package for details, but TLDR is following: every environmental variable prefixed by NG_ is put into object sharedEnvironment in /ui/apps/ui/src/environments/environment.generated.ts. As for the names NG_ prefix is removed and rest of the name is converted from UNDERSCORE_CASE into UnderscoreCase. Example: when only variable provided is NG_COLLECTIONS_PREFIX='beta_' then created environment.generated.ts will look as follows:

export const sharedEnvironment = {
  'collectionsPrefix': 'beta_'
}

export default sharedEnvironment;

sharedEnvironment environment is included in commonEnvironment from environment.common.ts. environment.generated.ts is generated in prestart and prebuild scripts which are automatically called when running npm start and npm run build

Install dependencies

npm i --force

Run

IMPORTANT!!! To change collections prefix copy <root>/.env file to <root>/backend with COLLECTIONS_PREFIX env variable.

npm start

Build

Build artifacts can be found in ui/dist/apps/ui. IMPORTANT!!! To change collections prefix copy <root>/.env file to <root>/backend with COLLECTIONS_PREFIX env variable.

npm build

Code automatic formatting

npx nx lint --fix

npx nx format:write

Solr

To run a standalone Solr instance run: docker-compose up -d solr1 zoo1. It registers all configsets from /solr/config/configsets/.

To access the web interface go to http://localhost:8983/solr.

To upload an additional configset:

docker run --rm -v "$PWD/solr/config/configsets:/configsets" --network=host solr:8.11 \
       solr zk upconfig -n "<name>" -d "/configsets/<name>" -z localhost:2181

To create a new collection, run

solr/create-collection.sh --name ess --config-name base-1

Obtain an example data sample using for example selects from https://docs.cyfronet.pl/display/FID/OpenAire+indexable+data.

Below assumes that Solr instance is available under localhost:8983, i.e. the port 8983 is forwarded to host network.

Solr schema

1) New schema have to be created for example using "Schema Designer" on the web GUI. 2) The schema should be placed in solr/config/configsets 3) Now schema is available for everyone. Use: docker compose build

Solr collection seeding

Add data to already existing collection

curl --location --request POST '<address>:<port>/solr/<collection_name>/update/json/docs' --header 'Content-Type: application/json' -T '<data_file>'

Running Recommender System locally

Check-out the code from https://git.man.poznan.pl/stash/scm/eosc-rs/online-ml-ai-engine.git (commit ea3545a6fc3 at the time of writing) to a separate directory, and build the image:

docker build -t "omae:0.0.1" .

Then, run the image:

docker run -p 9080:80 omae:0.0.1

Deployment

There are to be two machines:

API env variables:

General:

Operational:

Services:

Solr
STOMP
OIDC
Sentry
Other
Redirections

db envs:

The RS mock (online-ml-ai-engine, omae) should be deployed as described in "Running RS locally", with api's RS_URL pointed at it.

solr1 envs:

zoo1 doesn't have any envs.

DB migration

DB has to be migrated after code changes, to do it run alembic in the /backend/Dockerfile image, setting the DATABASE_URI env.

docker run --rm \
           --network "<ess-local_network>" \
           -e "DATABASE_URI=<as_for_api_above>" \
           -it $(docker build -q ./backend) \
           pipenv run alembic upgrade head

DB seed

Add the new dump content by running:

docker run --rm \
           --network "<ess-local_network>" \
           -e "DATABASE_URI=<as_for_api_above>" \
           -it $(docker build -q ./backend) \
           pipenv run python -m app.manager db seed-oag-2

Solr seed

Run solr1 with the explicit volumes as in docker-compose.yml, so that it creates missing configsets in Zookeeper on startup.

Then, you will need to create collections manually, see the section on Solr. The cluster should have collection: oag_researchoutcomes_prod_20211208_v2 (schema: oa_prod_20211208_v2).

Populate it using v2 publication data (see v2 comment to https://docs.cyfronet.pl/display/FID/OpenAire+indexable+data). The records are in publication/000550_0. They need to be transformed and pushed to the collection (see also the Solr section above for how to do it for v2).

Release

Releases are automatically created based on release-please-action.

Procedure:

  1. Run Trigger release
  2. Wait for automatic-release action in Actions , when action is finished, a new Pull Request should occur in Pull requests
  3. Go to Pull Request with name format: chore(master): release x.y.z, at this level you can customize release data:
    • release version is build based on x.y.z version in commit format chore(master): release x.y.z,
    • on x.y.z version change in commit, update also properly CHANGELOG.md section and version.txt
    • list of changes that will occur in release is written in CHANGELOG.md
    • the changes should land in one commit (squashed or amended) in format chore(master): release x.y.z
  4. Choose merge option Rebase and merge