New Search filters bring no results

[eu-genia]

The nice new search filters (search in specific fields etc) do not produce results as expected. E.g. Result

E.g. Result (only in MSS only in persName, not elsewhere!)

[PietroLiuzzo]

Please, detail. It looks to me your expectations are simply not matching the functionality. Where do you think you set "only in persName" ? please clarify what you are trying to search for. I have tried the following:

• search for "tewodros" in persons with occupation type ruler using facets
• search for tewodros in person-field
• search for persons with occupation type = ruler This does not return results, I am looking at this.

[PietroLiuzzo]

beside the above limited issue, relating to occupation type, I do not understand in your issue the relation between fields and other searches... screenshots clearly come from two different searches

[eu-genia]

in the example above, the results came only from manuscripts, not from other record types

If I search e.g. for

i get no results, I would expect quite many

[eu-genia]

yes sorry the examples above come from two different searches

• one for Tewodros + ruler in the search options hidden behind "+"
• the other one simply for Tewodros with the combination of fields hidden behind "="

[eu-genia]

Also in the search examples you showed above the results are only from manuscripts, not from other types of records. Or at least the persons of the name Tewodros would have appeared in the results I would expect...

[PietroLiuzzo]

I am afraid the issue with homily was yet another one. That I could find. at line 2319 titleStmt needs to be replaced by title to match the index name. I did this in the live version, but I could not do this change in the repository because when I looked at the diff of the files there where many changes in there which had not been committed to the repository.

[PietroLiuzzo]

@eu-genia your comments start from the assumption that the issue is with some sort of limitation to manuscripts. This is not the case. If you could please answer my question instead, that would help greatly. Please simply clarify what you are trying to search for.

[PietroLiuzzo]

The issue with the person field is in the index definition. <field name="person" expression="t:teiHeader//(t:person | t:personGrp)"/> this is an error, it should be <field name="person" expression="t:body//(t:person | t:personGrp)"/> the context being wrong, it does not match any person or personGrp.

This change will trigger a complete reindexing of the data. I suggest that this is done by you just after a reboot and when some time allowance without traffice / either off office working hours can be used.

• reboot as you prefer to do
• open monex, so you can monitor later
• open exide
• open the index configuration of the "expanded" collection ( /db/system/config/db/apps/expanded/collection.xconf)
• make the above change (line 17)
• save, when asked by exide if to reindex, say yes.
• monitor the reindexing progress in monex

This will take a while, because of the number of the indexes, collections and resources. The changed configuration file should be stored in the github repository.

[eu-genia]

Ok. Will do. Sorry, which changes have not been committed? I certainly committed to github changes in the modules I touched... I did not assume the search is only in mss but I only see results from mss, also in your Tewodros search example. None from a person type record, and they also have t:person. but i guess the correction above deals with it ))

[eu-genia]

I never touched queries.xqm - apart for static html display, now committed (https://github.com/BetaMasaheft/BetMas/commit/414240a01f9509ed6eee29a338e234dd0c17550c)

[eu-genia]

sorry before i change the xconf

• if i switch from teiHeaderto bodythat would mean that it does not bring the results it brings now - is this what we want? (or take the entire t:tei?)
• when I type body instead of teiHeader I get the warning message /db/system/config/db/apps/expanded/collection.xconf[http://www.w3.org/TR/xml-schema-1#UndeclaredPrefix?xs:string&xshttp://www.w3.org/TR/xml-schema-1#cvc-attribute.3?field&type&xs:string&QName](http://betamasaheft.eu:8080/db/system/config/db/apps/expanded/collection.xconf#461) - is this OK?

[eu-genia]

and

• I assume line 18 for places should also be changed accordingly to <field name="place" expression="t:body//t:place"/> ?

[eu-genia]

Some more search attempts:

I want to look for texts authored by John Chrysostom

I click

I get all possible works authored by anyone

the filter is not applied

[PietroLiuzzo]

• I assume line 18 for places should also be changed accordingly to <field name="place" expression="t:body//t:place"/> ?

yes, you are right.

The websites are not available since this morning: is it still indexing?

[eu-genia]

someone is bombarding it with queries... i rebooted yesterday night and it was all ok. this morning i found it unresponsive at 10, restarted exist, it was all ok between 10 at 16:00 when the monex which i checked on the way claimed 160 from 160 active db processes and all went in tilt. i restarted again now, but someone is very active, i did not make it today to check the logs.

[eu-genia]

(i made no changes and have not yet changed the conf, so it is not the indexing, it is someone's queries filling the memory i guess)

[PietroLiuzzo]

Where do you observe activity? in amplify? I see in monex a lot of hits, yes.

If you have not yet done so, I would suggest to

• turn off nginx (closes off all traffic)
• open nginx logs
• see if a bot is crawling
• edit the nginx conf to close it out
• restart nginx

let me know if you have done this before or you need help

[PietroLiuzzo]

There are also two active CVE for this repository. These need to be attended to.

[eu-genia]

I always have monex on, yes. I remember having a look at nginx conf when I had this problem last time (you remember I mentioned it when we talked) and saw that bots were already excluded by you, and the logs reported hits from the US and "Nigeria" judging by IPs; later I found out that around that time Ethiopian students were asked to present on Hamburg projects, so I assumed it were them via vpn. Then the activity disappeared all by itself, until this morning...

What can I do about CVE?

[PietroLiuzzo]

so, are you taking action on nginx? I am also ssh ed and looking at the log

[PietroLiuzzo]

I have cut off "AppleWebKit/537.36"

[PietroLiuzzo]

this is not ok, is closing off the rendering engine. the offending ip is this https://whatismyipaddress.com/ip/5.133.192.189

[PietroLiuzzo]

I have reverted that change, I think, with the current setting, we need to black list this ip. I have added the ip to blockips.conf

[PietroLiuzzo]

for the record

• sudo nano /var/log/nginx/access.log to check logs
• eventually ctrl + _ to go to a line
• eventually ctrl + v to go to last line
• check scrolling up
• identified frequent request with 200 response
• cd /etc/nginx/
• sudo nano nginx.conf
• checked for location of deny statments
• sudo nano blockips.conf
• added deny 5.133.192.189;
• ctrl + x to close, y to confirm
• sudo nginx -t test the configuration. if ok,
• sudo nginx -s reload

[eu-genia]

sorry, I was away from PC. thank you!

[PietroLiuzzo]

for the CVE I would start by reading them. they says there is an issue with a leaflet package. I have checked quickly for places where this may be used, found none and proceeded to remove the leaflet package in question in the two locations on the server where it was exposed. This may have consquences on the visualization of maps produced with leaflet which will have to be sorted out.

[PietroLiuzzo]

in facts there are plenty of issues in the html view of places... https://betamasaheft.eu/places/LOC3921Ifat/main although inspecting the page does not give any direct clue towards the lack of this specific module. the leaflet code may be changed and adapted accordingly. some css is really misbehaving on that page...

[eu-genia]

the peripleo map is "hidden" on purpose, because they seem to not serve the area any more, and only an empty gray square came up

[PietroLiuzzo]

It is not hidden at all... on chrome a squeezed line appears. remove the div in item.xqm or items.xql if you do not want it. you will also save the time of that query.

[PietroLiuzzo]

I am sure there was adequate discussions before each and all of the changes, like this one here, which have been made to the look of the application. Building that new style back in 2020 took a lot of work, discussions, documentation, decisions, etc. These are not the only places which are affected, homepage and search page almost made me cry when I saw them after a while, on Saturday. If I had the time I would look for those discussions and read them, I would be really curious to know what rationale drove these changes. If I may share some comments meant to be constructive the current home page, which is full of small untidinesses (see for example the "explore" button not in line, images not optimized requiring plenty of time to load, etc.) looks IMHO like a badly-designed tutorial and the search has lost its "simple" look in favour of a lengthy and cluttered text in a size which is hardly proportionate.

[eu-genia]

yes I did only this https://github.com/BetaMasaheft/BetMas/commit/39577f7b27e9bcc6dc409c7f3071d23c94b4147f squeezing the line I was not sure if I were to go on and suppress it completely as Solomon said he worked with it and it must stay, so the squeezed line remained as a link, but you can see even on going to peripleo directly that the area is not covered

[eu-genia]

Thank you I will try to optimize the homepage. (For me explore looks in line - it is the additional "=" button to the right that may seem not in place, but it stays there on scrolling.

Places: when I edited this https://betamasaheft.eu/resources/images/homepage/place.png all looked normal, without the overlapping boxes, and I have not touched anything related since, very strange, will try to look into it when have time

[eu-genia]

OK there was this issue https://github.com/BetaMasaheft/Documentation/issues/2027 main relations in item.xqm are not properly shown. I had closed the issue because I worked around in viewItemto satisfy the requests here https://github.com/BetaMasaheft/Documentation/issues/2013 but I did not find a solution to the bug itself

[eu-genia]

the xconf is updated and the indexing done; the homepage was slowed down by the big example screenshots, now hidden behind links, the small images are all optimized. (tested here) (it is faster than the previous homepage version)

[eu-genia]

cleaned index further https://github.com/BetaMasaheft/BetMas/commit/2006f5d3698713138bf1beb2c83bcc38a4de0620

still no results in search for tewodros in person-field search for persons with occupation type = ruler

[PietroLiuzzo]

I have tested this again with data and configuration on a newer version of exist db, perhaps this is just going to be solved with upgrade of exist version

[PietroLiuzzo]

because it works ok locally with 6.0.1