LA pipelines configuration and solr schemas should be updated

[vjrj]

I'm facing different issues trying to run the latest versions of pipelines in our LA portal (like issues with using a new name index, or missing fields in solr).

Things I've detected:

• the main configuration livingatlas/configs/la-pipelines.yaml (included in the debian package) should be updated probably using livingatlas/configs/la-pipelines-emr.yaml as a reference or a more recent one.
• solr schemas in livingatlas/solr/conf/ should be updated too. This is used by ala-install solrcloud role to create collections. For instance, currently this repository, has schemas different or outdated compared with these others.

Trying to find a correct more up-to-date la-pipelines.yaml I played with this patched configuration assuming that the -emr one is the more up-to-date version (but I'm not sure). Although this was not enough to solve, for instance, my name index issue.

la-pipelines.yaml.patch.txt

TIA

[adam-collins]

livingatlas/solr/conf/ is the source. Ideally there will not be copies elsewhere to maintain. Can you please list all of the missing fields so they can be investigated.

I am not familiar with the config yaml files but do note a surprisingly large number of differences. Some of the differences do not appear straight forward and I do not expect it to be resolved quickly.

[vjrj]

Hi @adam-collins , thanks for the fast response.

livingatlas/solr/conf/ is the source. Ideally there will not be copies elsewhere to maintain. Can you please list all of the missing fields so they can be investigated.

About solr config: Good to know that this is the source. I compared both schemas after _nest_parent_ was missing in my side, probably because I create the schema via ala-install some months ago. But as I found other differences in these schemas, and as the Helm part is quite updated, I was not sure about what schema and solr conf should be used.

About configs: I just compared livingatlas/configs/la-pipelines.yaml and livingatlas/configs/la-pipelines-emr.yaml assuming that the emr one is more updated and trying to find a working patch. It will be great if you only use a single reference config, but probably the emr one has a different format to inject their variables. Anyway, both configurations are quite different.