Not working after the last update to v26

ostasevych commented 1 year ago

After updating to the v26 I found that the search results are neither shown in UI, nor in the desktop client. There is no more context search block in the search results at all.

benjaminfrombe commented 1 year ago

+1

Happyfeet01 commented 1 year ago

Not working here:

`lltextsearch:check Full text search 26.0.0

Search Platform: Elasticsearch 26.0.0 (Selected) { "elastic_host": [ "http://internesuche:********@127.0.0.1:9200" ], "elastic_index": "internesuche", "fields_limit": "10000", "es_ver_below66": "0", "analyzer_tokenizer": "" }
Content Providers: Deck 1.9.0 [] Files 26.0.0 { "files_local": "1", "files_external": "0", "files_group_folders": "0", "files_encrypted": "0", "files_federated": "0", "files_size": "10", "files_pdf": "1", "files_office": "1", "files_image": "0", "files_audio": "0", "files_chunk_size": "2", "files_fulltextsearch_tesseract": { "version": "25.0.0", "enabled": "1", "psm": "4", "lang": "de,eng", "pdf": "1", "pdf_limit": "0" } }`

I hope anyone can maintain these App again.

`lars@livesystem-debian:~$ sudo -uwww-data php8.2 /var/www/nextcloud/occ fulltextsearch:test

.Testing your current setup: Creating mocked content provider. ok Testing mocked provider: get indexable documents. (2 items) ok Loading search platform. (Elasticsearch) ok Testing search platform. fail In Test.php line 304:

Search platform (Elasticsearch) down ?`

mahoneyr commented 1 year ago

Not sure if this is the same issue - with occ fulltextsearch:test I'm getting "No alive nodes. All the 1 nodes seem to be down." Have verified that I can browse to elasticsearch from externally and curl to it from server (I'm using a docker deployment and can curl from inside container).

lukas-staab commented 1 year ago

I have the same issue as the original author. I can curl elasticsearch instance and index, but NC Test does not work. With the 25 Fulltext Search / ES-Plugin on a 26 Nextcloud it worked (i added a manual exception to install), but since I upgraded the app to 26 it broke. I tried to manually downgrade by replacing the 26 code by the 25 code but failed to get Nextcloud running again. If this hints someone to a working workaround please let me know how you did it :)

lukas-staab commented 1 year ago

I tried again and got a working workaround. I got it running with the following versions:

Full text search - Elasticsearch Platform 25.0.0 (downgraded) Full text search 26.0.0 Full text search - Files 26.0.0 Full text search - Files - Tesseract OCR 25.0.0 (26 not yet released)

How I did it step by step:

deactivated ES Platform 26 App on NC webinterface
deleting / moving out the 26 App files like: mv ./nextcloud/apps/fulltextsearch_elasticsearch fulltextsearch_elasticsearch-26.back
Downloading v25: wget https://github.com/nextcloud-releases/fulltextsearch_elasticsearch/releases/download/25.0.0/fulltextsearch_elasticsearch-25.0.0.tar.gz (btw: this release is not listed on github, i just guessed the url based on the old ones)
unextract it: tar -xf fulltextsearch_elasticsearch-25.0.0.tar.gz
moving the v25 version in: mv fulltextsearch_elasticsearch ./nextcloud/apps/
activating the v25 app in the webinterface (you might have to confirm that you really want to activate it)

Hope it helps someone!

mahoneyr commented 1 year ago

Thanks - downgrading to v25 with these instructions worked for me also

ostasevych commented 1 year ago

Thanks - downgrading to v25 with these instructions worked for me also

Meanwhile, what to do to make the platform v26 working?

Happyfeet01 commented 1 year ago

hmmm not working for me @lukas-staab

mahoneyr commented 1 year ago

Is this actually an issue for fulltextsearch_elasticsearch?

I have it (temporarily) working by reverting to v25 and then reinstalling v26 through the app. Based on log files, I'm wondering if there's an issue with the files populating /var/www/html/custom_apps/fulltextsearch_elasticsearch/vendor/psr in v26?

Happyfeet01 commented 1 year ago

That fixed it for me ☺️https://github.com/R0Wi/elasticsearch-nextcloud-docker

ostasevych commented 1 year ago

That fixed it for me ☺️https://github.com/R0Wi/elasticsearch-nextcloud-docker

How? Is it working with the Full Text Search Platform version 26.0.0 in your case? As in my case it said failed. Which ES client version do you use?

Happyfeet01 commented 1 year ago

i use this Version.

PXL_20230604_082058113

ostasevych commented 1 year ago

i use this Version.

Great! As far as I understand your app version is 26.0.0 and the ES Client id ocker is v.8.6.1, am I right? Meanwhile, how did you manage to migrate your index?

Happyfeet01 commented 1 year ago

Yes, correct @ostasevych. I had no index anymore. I index the files for the first time.

ostasevych commented 1 year ago

Yes, correct @ostasevych. I had no index anymore. I index the files for the first time.

Meanwhile, what about security concerns? Is it so safe to trust to dockerised solution comparing to the legacy installed on the server side? I mean, might the index be accessible from outside in this case? What is your opinion?

Happyfeet01 commented 1 year ago

@ostasevych no security issue. Like the Version installed as service, both Versions listen to 127.0.01:9200. This means only to localhost. You can block this port for outgoing traffic with UFW.

ostasevych commented 1 year ago

Yes, correct @ostasevych. I had no index anymore. I index the files for the first time.

Need some help here. I have a lot of files, and the indexing process is quite long, so I've started it from remote ssh terminal several times due to drop/downs.

So, the curl shows that the data are indexed

curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "query_string" : {
            "query" : "test",
            "default_field" : "title"
        }
    }
}
'

But the reply of the command:

sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search userA test is just search

and no data in the webUI :(

So, how to check whether it works?

UPD After restarting the indexing I am getting tons of unknown errors:

Are any tools or methods to debug fulltextsearch indexing and querying?

lukas-staab commented 1 year ago

this could be the underling error: https://github.com/nextcloud/fulltextsearch_elasticsearch/issues/256

@Happyfeet01 are you mayben not using basic auth with your docker setup? i did before i downgraded (also did not use docker)

Happyfeet01 commented 1 year ago

No basic Auth. Only Cluster Name and Address. But your linked issue seems the reason.

Before I tried this Docker Method I using the auth for Elasticsearch.

ostasevych commented 1 year ago

Again about reindexing. I am still struggling to make the full text search working on my instance:

NC 26.0.3 (bare metal)
All fulltextsearch apps are update
Using groufolders app
Elasticsearch via dockerised container: 8.6.1

Reindexing has been done with the following commands:

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:stop
$ curl -X DELETE localhost:9200/my-index
$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:reset
$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:index

Searching with curl in index "my-index" for the word "Ivanova" (our member) gives (user data are stripped and scratched):

$ curl -X GET "localhost:9200/my-index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "title": "Ivanova"
    }
  }
}
'
{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 9.4341955,
    "hits" : [
      {
        "_index" : "my-index",
        "_id" : "files:6811",
        "_score" : 9.4341955,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user1",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova.pdf",
          "users" : [ ],
          "content" : "Ivanova document"
          "tags" : [ ],
          "attachment" : {
            "date" : "2022-04-06T06:29:11Z",
            "content_type" : "application/pdf",
            "format" : "application/pdf; version=\"A-2b\"",
            "modified" : "2022-04-06T06:29:11Z",
            "language" : "uk",
            "creator_tool" : "ABBYY FineReader 14",
            "content_length" : 1612
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanov.pdf",
            "user1" : "Scanned/Ivanov.pdf",
            "user2" : "GroupA/Scanned/Ivanov.pdf",
            "user3" : ""
          },
          "hash" : "aeca335860b2f59954q5e7fd34b174a1"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:6812",
        "_score" : 9.4341955,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova2.pdf",
          "users" : [ ],
          "content" : "Ivanova........................................................... ...................",
          "tags" : [ ],
          "attachment" : {
            "date" : "2022-04-01T08:00:50Z",
            "content_type" : "application/pdf",
            "format" : "application/pdf; version=\"A-2b\"",
            "modified" : "2022-04-01T08:00:50Z",
            "language" : "uk",
            "creator_tool" : "ABBYY FineReader 14",
            "content_length" : 1260
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanova2.pdf",
            "user1" : "Scanned/Ivanova2.pdf",
            "user2" : "GroupA/Scanned/Ivanova2.pdf",
            "user3" : "",
          },
          "hash" : "e09e889376ebe62b907b8023f37d21a9"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:1576",
        "_score" : 9.30954,
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivaniva CV.pdf",
          "users" : [ ],
          "content" : "",
          "tags" : [ ],
          "attachment" : {
            "date" : "2022-06-02T07:57:28Z",
            "keywords" : "Scanned image",
            "content_type" : "application/pdf",
            "author" : "NAPS2",
            "format" : "application/pdf; version=1.4",
            "modified" : "2022-06-02T07:57:28Z",
            "language" : "lt",
            "title" : "Scanned image",
            "creator_tool" : "NAPS2",
            "content_length" : 4
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Scanned/Ivanova CV.pdf",
            "user5" : "GroupA/Scanned/Ivanova CV.pdf",
            "user3" : ""
          },
          "hash" : "53f4a218a7ac0f31648efc0834c35199"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:518673",
        "_score" : 8.218001,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova Bulletin.docx",
          "users" : [
            "user8"
          ],
          "content" : "Ivanova Bulletin text text",
          "tags" : [ ],
          "attachment" : {
            "date" : "2023-06-13T09:19:00Z",
            "content_type" : "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
            "author" : "Administrator",
            "modifier" : "User6",
            "modified" : "2023-06-14T13:32:00Z",
            "language" : "uk",
            "content_length" : 1701,
            "print_date" : "2023-06-14T13:30:00Z"
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Bulls/Ivanova Bull.docx",
            "user9" : "GroupA/Bulls/Ivanova Bull.docx"
          },
          "hash" : "99a498dbb1db6f68a6be3793e30a9476"
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:6806",
        "_score" : 7.622202,
        "_source" : {
          "share_names" : {
            "user0" : "GroupA/Scanned/Photos/Ivanova",
            "user2" : "Scanned/Photos/Ivanova",
            "user3" : ""
          },
          "owner" : "user0",
          "users" : [ ],
          "groups" : [
            "GroupA",
            "GroupB",
            "GroupC",
            "GroupD"
          ],
          "circles" : [ ],
          "links" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "subtags" : [ ],
          "tags" : [ ],
          "hash" : "",
          "provider" : "files",
          "source" : "files_group_folders",
          "title" : "GroupA/Scanned/Ivanova",
          "parts" : [ ],
          "content" : ""
        }
      },
      {
        "_index" : "my-index",
        "_id" : "files:466569",
        "_score" : 5.747198,
        "_ignored" : [
          "content.keyword"
        ],
        "_source" : {
          "owner" : "user0",
          "groups" : [
            "GroupD"
          ],
          "circles" : [ ],
          "metatags" : [
            "files_group_folders"
          ],
          "source" : "files_group_folders",
          "title" : "GroupA/Applications/Ivanova.docx",
          "users" : [
            "user4",
            "user5",
            "user6",
            "user7"
          ],
          "content" : "Application of Ivanova text text text",
          "tags" : [ ],
          "attachment" : {
            "date" : "2023-03-10T12:25:00Z",
            "content_type" : "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
            "author" : "user5",
            "modifier" : "user10",
            "modified" : "2023-05-17T17:55:09Z",
            "language" : "uk",
            "content_length" : 1550
          },
          "provider" : "files",
          "subtags" : [ ],
          "parts" : {
            "comments" : ""
          },
          "links" : [ ],
          "share_names" : {
            "user0" : "GroupA/Applications/Ivanova.docx",
            "user6" : "GroupA/Applications/Ivanova.docx",
            "user3" : "",
          },
          "hash" : "f8c12747f9b240ed696385d2aff7f0fe"
        }
      }
    ]
  }
}

Next, I search for Ivanova on behind of a user0

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user0 Ivanova
search
> Files
 - 518673 score:0
 - 480179 score:0
 - 514585 score:0
 - 527182 score:0
 - 531276 score:0
 - 363692 score:0

The documents are stored in the groupfolder, which is accessible to user0, user1, user2, user3, user10, but that's what I see when searching with the help of NC fulltextsearch app:

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user10 Ivanova
search
> Files

$ sudo -u www-data php /var/www/nextcloud/occ fulltextsearch:search user3 Ivanova
search
> Files
 - 97091 score:0

Obviously, users in webUI cannot see the proper output.

Would you be so kind to explain, why it is not working as expected? What should I do to make it working? Thanks!

lukas-staab commented 1 year ago

Are you aware that Elasticsearch 8 is not offical supported? I would start there. see https://github.com/nextcloud/fulltextsearch_elasticsearch

meonkeys commented 1 year ago

Or is it? https://github.com/nextcloud/fulltextsearch_elasticsearch/wiki#installation-of-elasticsearch-via-docker points to https://github.com/R0Wi/elasticsearch-nextcloud-docker, which suggests elasticsearch version 8.6.1.

ostasevych commented 1 year ago

Or is it? https://github.com/nextcloud/fulltextsearch_elasticsearch/wiki#installation-of-elasticsearch-via-docker points to https://github.com/R0Wi/elasticsearch-nextcloud-docker, which suggests elasticsearch version 8.6.1.

Exactly!

lukas-staab commented 1 year ago

Are you aware that Elasticsearch 8 is not offical supported? I would start there. see https://github.com/nextcloud/fulltextsearch_elasticsearch

There was an update to the Readme. Seems like ES-plugin 26+ supports ES 8. If the 26 plugin does not work that means you still need to use 7, otherwise 8 is (newly) working.

Also the update to ES 8 might solve some issues people had here when going up to ES 26 plugin

overcuriousity commented 1 year ago

Tried it yesterday, it did not work with ES plugin 26. and elasticsearch 8.6.1 or newer. Maybe it changed with nc version 27.0.2, which I downloaded a couple hours ago. not tested yet.

nextcloud / fulltextsearch

Not working after the last update to v26 #746