search

usaybia / srophe-eXist-app

eXist code for Syriaca.org: The Syriac Reference Portal
GNU General Public License v3.0
1 stars 0 forks source link

Fix REST API #75

Closed nathangibson closed 3 years ago

nathangibson commented 3 years ago

We want to use this to improve our tagging workflow, see https://github.com/usaybia/usaybia-data/issues/98.

At this point, I don't think we need to change any of the parameters used by Syriaca, except that collection is irrelevant for Usaybia.

wsalesky commented 3 years ago

This should work on redeploy. When merged into master I will cherry-pick this change into development.

nathangibson commented 3 years ago

I might not get back to this until Monday, but ...

Can you help me troubleshoot the Rest API for the dev server? It is returning a JSON response, but with "0 results" for everything I try, e.g. https://dev.usaybia.net/api/search/persName?q=Muhammad

I realized I don't think the JSON Parser & Serializer app was installed, so I installed that. Then I thought maybe I needed to reregister the API, so I ran the script you sent me earlier in exide. Same thing.

Is it an indexing issue?

nathangibson commented 3 years ago

The background is I'm trying to get your diacritics changes to the API working.

wsalesky commented 3 years ago

Looks to me like the dev server needs to be index. Run the modules/index.xql

nathangibson commented 3 years ago

@wsalesky That did it, thanks! It is supposed to be indexed on installation into eXist, right? Did it need to be reindexed because of my installing the JSON Parser app or do you think it's a problem with the Docker image build/deploy process?

wsalesky commented 3 years ago

@nathangibson The way the application is set up now (because of how eXist 5.0 handles faceting) the configuration file in the data directory is empty, and you build it from the application. This means you have to run the index.xql on deployment. This may not have been the best way to handle indexing, especially for your workflow. What you may want to do, is build the .xconf file by running index.xql and then store it in the usaybia-data repository, that way your data will get index on deployment, you will just need to rerun index.xql and update the .xconf file when you update your facets, or change how you are indexing files.

nathangibson commented 3 years ago

@wsalesky Interesting. In that scenario would we also need to reindex whenever we modify the data?

Also, if we store the .xconf file in usaybia-data, does it get loaded into the right place in eXist, or do I need to copy it somewhere on deployment?

wsalesky commented 3 years ago

If you store the generated .xconf into usyabia-data you would only need to run the usaybia/modules/index.xql if you change what is being indexed, such as adding new fields or custom facets, or range indexes. Otherwise it will index new data as it is added. This is probably the best way going forward.

nathangibson commented 3 years ago

I'm looking in exide to see where the xconf file is. The only ones I can find are collection.xconf

nathangibson commented 3 years ago

But those are only a few lines

nathangibson commented 3 years ago

Is it /db/system/config/db/apps/usaybia-data/collection.xconf ?

wsalesky commented 3 years ago

@nathangibson This one: https://github.com/usaybia/usaybia-data/blob/master/collection.xconf

Would need to be replaced with the one generated by modules/index.xql, which after you run will be stored in /db/system/config/db/apps/usaybia-data/collection.xconf

wsalesky commented 3 years ago

The one in the db is automatically generated, in order to build the facets correctly.

nathangibson commented 3 years ago

Got it. Will https://github.com/usaybia/srophe-eXist-app/blob/2b526e850fb8a25ff5ec859644378d55c877d6ef/pre-install.xql#L32 then automatically copy it to the right location?

wsalesky commented 3 years ago

No, sorry these are not the same thing. Let me try to outline how it works more clearly.

  1. Each .xar pacakage has a .xconf file.

  2. On deployment the .xconf file in the application is copied for you to the appropriate system/config directory. a. This means that usaybia/collection.xconf is copied to '/db/system/config/usaybia/collection.xconf' We do not index anything in the app, because there is no data there, so this xconf file is mostly empty. b. And usaybia-data/collection.xconf is copied to '/db/system/config/usaybia-data/collection.xconf' This is the important configuration file, it sets up full text indexes, range indexes, and now facets and fields.

  3. Because facets are defined in the application, I have added a function that checks all the facet definition files and automatically builds a matching index configuration to add the specified facets to the index. (Previously facets were not specified in the index, now they are handled by Lucene, which should make them faster, but adds some complexity to indexing). This function is run by the usaybia/module/index.xql xquery, it checks all facet definition files and any predefined fields and builds a new collection.xconf file which it then copies over to the data directory (/db/system/config/usaybia-data/collection.xconf) and triggers a re-index.

  4. To insure that your data is indexed on deployment you can simply run usyabia/modles/index.xql, and then open the generated collection.xconf (/db/system/config/usaybia-data/collection.xconf) and save it to your data repository. This will only need to be updated as you change facets, or add fields to your index.

Hope that helps clarify. There may be a simpler solution, I just wanted to make adding facets as simple as it had been in the past.

nathangibson commented 3 years ago

Thanks, this is clear now! I was confusing the pre-install files from the two apps. I guess it is actually https://github.com/usaybia/usaybia-data/blob/6c39bbf80bf400640c4713c162b2c21bd5cfec7f/pre-install.xql#L32 that will copy it to the correct place.