This is the repository for the Talent Catalog (TC), which manages data for refugees looking for skilled migration pathways into safe countries and employment.
This repository is a "mono-repo", meaning it contains multiple sub-modules all of which make up the Talent Catalog system. In particular, it contains:
/api/candidate
provided by the server./api/admin
provided by the server./api/public
provided by the server.Contributions are very welcome. Please see our contribution guidelines. They should be submitted as pull request.
IMPORTANT NOTE:
These instructions are tailored for Mac users using Intellij, as this is what we use for development.
On a Mac, installing with Homebrew usually works well. eg "brew install xxx".
It is also probably easier to install Java directly (or from your development IDE - see below) rather than using brew.
Download and install the latest of the following tools.
Homebrew - Homebrew website
IntelliJ IDEA - Intellij website
Java 17
The current version of Java supported is Java 17. We use the Temurin release (however there should be no issues using other releases). One way (but you can choose whichever method you like) to manage Java versions is with sdkman. A .sdkmanrc file exists when you check out the repository. You can get sdkman by running the following:
curl -s "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"
sdk install 17.0.11-tem
Intellij will load the JDK through the .sdkmanrc file.
Update the Project SDK:
IntelliJ Settings:
-parameters
to theAdditional command line parameters
textbox.Project bytecode version
to match the JDK chosen (e.g. 17).Code Style
Gradle https://gradle.org/install/
brew install gradle
See https://www.jetbrains.com/help/idea/angular.html and https://nodejs.org/en/about/releases/
brew install node@16
Angular CLI https://angular.io/cli
npm install -g @angular/cli@16
Docker
Elasticsearch (for text search)
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.17.11
Kibana (for monitoring Elasticsearch)
docker pull docker.elastic.co/kibana/kibana:7.17.11
Git - see Git website - Not really necessary now with Intellij which will prompt you install Git if needed
PostgreSQL - Postgres website
brew install postgresql@14
brew services restart postgresql@14
These tools do not need to be installed in order to get the code up and running on your development
machine. However, they are needed if you want to build the TC's AWS cloud infrastructure
from the Terraform definitions in the infra
folder.
Once installed, needs to be configured. Log in to your AWS account, click on user top right, select Security Credentials, create access key, then download to CSV file. Then, theoretically this should work
aws configure import --csv path-to-downloaded-file.csv
...but it doesn't currently (it fails saying that it is missing a User Name header). Instead, just run this and manually copy/paste the values from the CSV file as prompted.
aws configure
brew install terraform
Once installed, you can run Terraform from the directory containing your main Terraform
file (main.tf).
In order to populate "secret" configuration values that Terraform needs to set up as
environment variables for the TC software, you need to copy a special file terraform.tfvars
to that directory before running terraform. Contact support@talentcatalog.net for a copy of that
file.
Then you can run init
(only need to do this once), and then plan
or apply
, as needed.
terraform init
terraform plan
terraform apply
Use the psql tool.
psql postgres
Now you will see the command line prompt =#
CREATE DATABASE tctalent;
CREATE USER tctalent WITH SUPERUSER PASSWORD 'tctalent';
\q
Ask Talent Catalog developers for a pg_dump
of the database. Note that the dump does not have to
be recent.
The Talent Catalog software will automatically apply any requuired updates to the database
definition, driven
by Flyway files stored in GitHub. A standard dump file is kept specifically for getting new
developers started.
Just ask for a copy of the file.
pg_dump --file=path/to/file.sql --create --username=tctalent --host=localhost --port=5432
Use psql
to import that dump file into your newly created database.
psql -h localhost -d tctalent -U tctalent -f path/to/file.sql
Use docker
to run from Docker desktop for Mac, or (replacing appropriate version number)...
docker rm elasticsearch
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.17.11
Elasticsearch will run listening on port 9200. You can verify this by going to localhost:9200 in your browser
Can run from Docker desktop for Mac, or (replacing appropriate version number)...
docker rm kibana
docker run --name kibana --link elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:7.17.11
Kibana runs listening on port 5601. You can verify this by going to localhost:5601 in your browser
Some secret information such as passwords and private keys are set in environment variables - including programmatic access to Talent Catalog's Amazon AWS, Google and Salesforce accounts. If these environment variables are not set the application will fail at start up. Contact other Talent Catalog developers for a copy of a "secrets" file suitable for developers. On development computers you hook the file, tc_secrets.txt, into your computer's start up to set the relevant environment variables. For example add "source ~/tc_secrets.txt" to .bash_profile or .zshenv depending on whether you are running bash or zsh.
Create a new Run Profile for org.tctalent.server.TcTalentApplication
.
In the Environment Variables section of Intellij, check the
"Include system environment variables" checkbox.
Run the new profile, you should see something similar to this in the logs:
Started TcTalentApplication in 2.217 seconds (JVM running for 2.99)
The "Candidate Portal" is an Angular Module and can be found in the directory
talentcatalog\ui\candidate-portal
.
Before running, make sure all the libraries have been downloaded locally by running npm install
from the root directory of the module (i.e. talentcatalog\ui\candidate-portal
):
cd talentcatalog\ui\candidate-portal
npm install
It is also a good idea to install fsevents for MacOS which will greatly reduce your CPU usage
npm install fsevents
npm rebuild fsevents
Then from within the same directory run:
ng serve
You will see log similar to:
chunk {main} main.js, main.js.map (main) 11.9 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 236 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 6.08 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 16.6 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 3.55 MB [initial] [rendered]
i 「wdm」: Compiled successfully.
The Candidate Portal is now running locally and you can open a browser (chrome preferred) to:
Note: this is for development mode only. In production, the Candidate Portal module will be bundled into the server and serve through Apache Tomcat.
The "Public Portal" is an Angular Module and can be found in the
directory talentcatalog\ui\public-portal
.
As for the "Candidate Portal", make sure all libraries are installed locally.
Then from within the same directory run:
ng serve
You will see log similar to:
chunk {main} main.js, main.js.map (main) 11.9 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 236 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 6.08 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 16.6 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 3.55 MB [initial] [rendered]
i 「wdm」: Compiled successfully.
The Public Portal is now running locally and you can open a browser (chrome preferred) to:
Note: this is for development mode only. In production, the Public Portal module will be bundled into the server and serve through Apache Tomcat.
The "Admin Portal" is an Angular Module and can be found in the directory
talentcatalog\ui\admin-portal
.
As for the "Candidate Portal", make sure all libraries are installed locally.
Then from within the same directory run:
ng serve
You will see log similar to:
chunk {main} main.js, main.js.map (main) 11.9 kB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 236 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 6.08 kB [entry] [rendered]
chunk {styles} styles.js, styles.js.map (styles) 16.6 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 3.55 MB [initial] [rendered]
i 「wdm」: Compiled successfully.
The Admin Portal is now running locally and you can open a browser (chrome preferred) to:
Note: this is for development mode only. In production, the Admin Portal module will be bundled into the server and serve through Apache Tomcat.
SystemAdmin
and password password
that can be used to log in to the admin portal in development.org/talentcatalog/server/configuration/SystemAdminConfiguration.java
esload
Note that you have to separately upgrade each of the Angular directories:
Assuming that the package.json in each of the above directories has the right versions already in there you just need run the following commands in each directory.
npm install
Note and fix any errors. "npm outdated" is good for identifying outdated libraries "npm update --save" will update versions to the latest version within the allowed versions specified by the package.json.
Once all versions are updated for the current version of Angular, you can run the Angular update as follows.
ng update
This will prompt you to update the Angular core and cli. For example:
ng update @angular/core@13 @angular/cli@13
This will update package.json with the appropriate Angular versions which will drive updates of other dependent libraries.
You may find that you need to manually upgrade versions of some tools in package.json so that they work with the new version of Angular. For example, you might need to upgrade the version of ng-bootstrap to a version that works with the later version of Angular. Look at the doc of the library in question to select the correct version
You may also need to make changes to your Angular code because of changes in Angular, or because of changed APIs in the dependent libraries.
See https://stackoverflow.com/questions/11284634/upgrade-node-js-to-the-latest-version-on-mac-os
We use GitHub. Our repository is called talentcatalog - https://github.com/Talent-Catalog/talentcatalog
See the GitHub wiki for additional documentation.
The main branch is "master". We only merge and push into "master" when we are ready to deploy to production (rebuild and upload of build artifacts to the production environment is automatic, triggered by any push to "master". See Deployment section below).
Master should only be accessed directly when staging is merged into it, triggering deployment to production. You should not do normal development in Master.
The "staging" branch is used for code which is potentially ready to go into production. Code is pushed into production by merging staging into master and then pushing master. See Deployment section below.
Staging is a shared resource so you should only push changes there when you have finished changes which you are confident will build without error and should not break other parts of the code.
As a shared resource, staging is the best way to share your code with other team members to allow them to merge your code into their own branches and also to allow them to review your code and help with testing.
Rebuild and upload of build artifacts to the AWS testing environment is automatic when any push is made to "staging".
New development should be done in branches.
Typically, you should branch from the staging branch, and merge regularly (eg daily) from staging so that your code does not get too far away from what everyone else is doing.
When you are ready to share your code for others to take a look at and for final joint testing and eventual deployment, merge your branch into staging.
On your branch you should commit often - doing separate commits for specific functionality, rather than lumping different kinds of functionality into a single big commit. That makes commits simpler to review and understand. It also makes it easier to revert specific functionality when you have got something wrong and decide to start again, doing it differently.
You should feel comfortable pushing regularly - often doing Commit and Push at the same time. Pushing is effectively saving your work into the "cloud" rather having changes just saved on your computer.
See the Deployment and Monitoring pages on the GitHub wiki.