Adding for developer testing

[qjhart]

Adding Experts for Development testing

These are some developer notes on testing particular experts for Aggie Experts

Adding Pre-Calculated Experts

Currently, developers typically get a small set of experts included in their testing setup. The methodology for this is that there is a pre-defined GCS bucket containing a small number of experts. On initialization, the contents of this bucket is added to the system.

One can see the configuration parameter with bin/aggie-experts config. The buckets are available for viewing via the google cloud console, or via tools like:

gsutil ls gs://aggie-experts-fcrepo-dev/expert

Fetching a single expert

In the aggie expert filesystem, an Experts' relationships to citations, and grants, are stored within the directory of the expert themselves. This means that a developer can get all items relevant for a user via one directory. For example, you can use gsutil to get a single user (like say Jonathan Eisen, 226d2dccfba4c2be04aedd4f9f942e42) . You can get that from any other bucket as well. For example. the command below gets eisen's data from the sandbox bucket.

d=test
mkdir ${d}
gsutil -m rsync -r -x '(?!expert/226d2dccfba4c2be04aedd4f9f942e42.*)' gs://aggie-experts-fcrepo-sandbox ${d}

The -x '(?!expert/226d2dccfba4c2be04aedd4f9f942e42.*)' parameter excludes everything that isn't part of eisen's record.

Adding with fin

After fetching your data, you can easily add it to your development for testing using fin

fin config set host http://localhost
fin auth login
fin io import --import-from-root --fcrepo-path-type=subpath --log-to-disk ${d}

You can watch the import fire events in the 'http://localhost/fin/admin/' dashboard. This is an example the above import

Creating new Experts

In addition to loading Experts that have already been created. You may actually be testing a new harvest methodology. In this case, you want to create your experts locally, and test the system with those records before adding them to a cloud location. You can do all this in the fuseki container of your development system.

Say for example, you want to test all users in fcrepo-sandbox. This is cdl group 1587. You can create these experts like:

# in dc exec fuseki /harvest-entrypoint.sh bash
group=1587
experts cdl --cdl.groups=${group} --debug-save-xml --author-trim-info --fuseki.delete --output=fcrepo-sandbox

When complete you now have a directory ~fcrepo-sandbox~ with these users. You can use ~fin~ to import these, but there is one trick. In development, fuseki thinks that the gateway (and hence fcrepo) is at: http://gateway:3001, while your browser thinks it's at http://localhost. You could modify your ~/etc/hosts~ file, and match ports, or you can tweak the way you get your token.

     fin config set host http://gateway:3001
     fin auth login --headless

That responds with:

Visit this URL on any device to log in, then paste token below.
http://gateway:3001/auth/keycloak-oidc/login?cliRedirectUrl=http%3A%2F%2Fgateway%3A3001%2Fauth%2Flogin-shell&provideJwt=true&force=true

Modify that URL to

http://localhost/auth/keycloak-oidc/login?cliRedirectUrl=http%3A%2F%2Flocalhost%2Fauth%2Flogin-shell&provideJwt=true&force=true

and paste that into ~fin~ You can then add to your system with:

fin io import --fcrepo-path-type=subpath --import-from-root fcrepo

[qjhart]

@rakunkel-ucd @UcDust Please review, and if it it looks good, then remove yourselves from the issue.

[UcDust]

@qjhart

Thanks, this worked perfectly!

I assume if we leave off the -x param in the gsutil export step, it would include everything? Any risk there?

[qjhart]

@UcDust - No risk at all! You could also add 5 users into the same test directory. You could also remove all but a few relationships, or alter the data if you wanted to try some weird testing.

Also, if you do sync the whole sandbox over, fin will check and only import changes if you try to import the whole set mulitple times.