problem with samba , search not working on index

[Beleggrodion]

Hi,

I don't know where to ask this, but because i use your tool to index the samba server. Perhaps the issue, which i have, is known and perhaps a solution is already exists. I'm not an mac expert, but the users use macs.

First, all worked as expected at the beginning. Then we had some issue's on some Mac's that Finder Frezzes on copy big folder trees and we do some setting changes in de smb.conf and after that was fixed, we detected that on all mac's the search don't work anymore. I don't changed anything on the elasticsearch parameters.

The Index shoul'd be run as expected because " fs2es-indexer search" gives me the example file, and direct on the server via samba i receive the file mdsearch 192.168.x.x testshare 'kMDItemFSName=="test1234.txt"' -U test

But when i try it on the mac with mdfind -onlyin /Volumes/testshare test123.txt i receive an empty result.

The only what i see in the log is the following error:

[2022/11/25 15:04:46.238474,  4, pid=3996626, effective(10023, 513), real(10023, 0)] ../../source3/smbd/vfs.c:939(vfs_ChDir)
  vfs_ChDir to /data/sasData1/shares/testshare
[2022/11/25 15:04:46.238752,  0, pid=3996626, effective(10023, 513), real(10023, 0)] ../../source3/rpc_server/mdssvc/es_mapping.c:182(es_map_sl_attr)
  es_map_sl_attr: No JSON type mapping for [com_apple_mail_attachmentNames]
[2022/11/25 15:04:46.238841,  0, pid=3996626, effective(10023, 513), real(10023, 0)] ../../source3/rpc_server/mdssvc/mdssvc.c:977(slrpc_open_query)
  slrpc_open_query: backend search_start failed

I read the "Debugging the search" part, and see the row

If this fails: check your samba-logs on the server. Any entries with "rpc_server", "mds" or "mdssvc" in it?

But i don't see what is to test or try when this happens? Because the mapping file exists and is set under /usr/share/samba/mdssvc/elasticsearch_mappings.json and is the original file which is provided by samba.

I use samba 4.13.17 under Ubuntu 20.04.5 LTS and the affected Mac's have High Sierra (hardware to old for newer os) or Monterey. Elastic Search is 8.5.0. (As Docker Image in the Same Server)

And yes on the client "mdutils -s /Volumes/testshare says "Server search enabled".

[MatthiasKuehneEllerhold]

The mdfind on the mac doesnt find anything? Then something is wrong in between samba and the ES instance.

Can you check your smb.conf? Did you try the debugging steps from the README?

[Beleggrodion]

I don't know what i did then, it works now since some months (i don't know if elastic search is used or not but client find files and thats important) after i tried with some parameters in samba. bellow a snipped of the current config.

i tried the debugging steps from the README as i know.

[global]
        workgroup = DOMAIN
        realm = DOMAIN.INTRA
        netbios name = srv-data

        server string = Ubuntu Server
        interfaces = 192.168.123.10/24
        server role = classic primary domain controller
        keep alive = 10
        server signing = off

        log level = 4
        log file = /var/log/samba/%m.log
        max log size = 5000
        debug pid = yes
        debug uid = yes
        utmp = yes

        security = user
        domain logons = yes
        os level = 64
        logon path = \\srv-data\profiles\%U
        logon home = \\srv-data\%U
        logon drive = H:
        logon script =

        passdb backend = ldapsam:"ldap:///lin-srv01.domain.intra"
        ldap ssl = off
        ldap admin dn = cn=admin,dc=domain,dc=intra
        ldap delete dn = no

        ntlm auth = yes

        ## Sync UNIX password with Samba password
        ## Method 1:
        ldap password sync = yes
        ## Method 2:
        ;ldap password sync = no
        ;unix password sync = yes
        ;passwd program = /usr/sbin/smbldap-passwd -u '%u'
        ;passwd chat = "Changing *\nNew password*" %n\n "*Retype new password*" %n\n"

        ldap suffix = dc=domain,dc=intra
        ldap user suffix = ou=Users
        ldap group suffix = ou=Groups
        ldap machine suffix = ou=Computers
        ldap idmap suffix = ou=Idmap

        idmap config * : range = 10000 - 20000

        add user script = /usr/sbin/smbldap-useradd -m '%u' -t 1
        rename user script = /usr/sbin/smbldap-usermod -r '%unew' '%uold'
        delete user script = /usr/sbin/smbldap-userdel '%u'
        set primary group script = /usr/sbin/smbldap-usermod -g '%g' '%u'
        add group script = /usr/sbin/smbldap-groupadd -p '%g'
        delete group script = /usr/sbin/smbldap-groupdel '%g'
        add user to group script = /usr/sbin/smbldap-groupmod -m '%u' '%g'
        delete user from group script = /usr/sbin/smbldap-groupmod -x '%u' '%g'
        add machine script = /usr/sbin/smbldap-useradd -w '%u' -t 1

        map to guest = bad user

        local master = yes
        os level = 33
        preferred master = yes

        wins support = yes
        dns proxy = yes

        max open files = 20000

        ## Global Share Config
        vfs objects = acl_xattr catia fruit streams_xattr
        #fruit:metadata = stream
        fruit:model = MacSamba
        fruit:posix_rename = yes
        fruit:veto_appledouble = no
        fruit:zero_file_id = yes
        #fruit:nfs_aces = no
        fruit:wipe_intentionally_left_blank_rfork = yes
        fruit:delete_empty_adfiles = yes
        fruit:aapl = yes

        fruit:ressource = xattr
        #fruit:encoding = native
        fruit:locking = none

        fruit:copyfile = no

        ;;fruit:locking = netatalk
        ;streams_xattr:store_stream_type = no (Windows Problem)
        ;streams_xattr:prefix = user (Windows Problem)

        ea support = yes

        hide files = /.DS_Store/Network Trash Folder/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary Items/.TemporaryItems/.VolumeIcon.icns/Icon?/.FBCIndex/.FBCLockFolder/

        spotlight backend = elasticsearch
        elasticsearch:address = localhost
        elasticsearch:index = files
        elasticsearch:port = 9200
        spotlight = yes

        rpc_daemon:mdssd = disabled
        #rpc_server:mdssvc = embedded
        rpc_server:mdssvc = disabled

        smb encrypt = off

        use sendfile = yes
        aio read size = 0
        aio write size = 0
        strict sync = yes
        sync always = no

## Netlogon Services
[netlogon]
        path = /var/lib/samba/netlogon
        comment = Domain logon service
        browseable = no

## Roaming Profiles
[profiles]
        path = /var/lib/samba/profiles
        browseable = no
        writeable = yes
        guest ok = yes
        create mask = 0611
        directory mask = 0700
#       profile acls = yes
        csc policy = disable
        map system = yes
        map hidden = yes
        hide files = /desktop.ini/outlook*.lnk/*Briefcase*/

[install]
        path = /data/shares/install
        comment = Software Repository
        browsable = yes
        public = yes
        writeable = yes

[homes]
        #path = /data/shares/users
        comment = Heimatverzeichnisse
        browsable = no
        writable = yes

## Drucker Freigaben
#[printers]
#       path = /var/spool/samba
#       comment = Drucker
#       printable = yes
#       guest ok = no
#       public = no
#       writable = no
#       browsable = no

#[print$]
#       path = /data/shares/print
#       comment = Druckertreiber
#       browsable = yes
#       guest ok = no
#       read only = yes
#       write list = root, @Domain-Admins

## DOMAIN Firmen Freigaben
[Verkauf]
        path = /data/shares/Services/Verkauf
        vfs objects = acl_xattr fruit streams_xattr
        writeable = yes
        browseable = yes
        public = no
        create mode = 0777
        directory mode = 0777
        force create mode = 0777
        force directory mode = 0777
        valid users = root, @Verkauf
        force user = root
        force group = Verkauf
        write list = root, @Verkauf
        veto files = /._*/.DS_Store/*Thumbs.db*/.Trashes/.TemporaryItems/
        delete veto files = yes

[SD]
        path = /data/shares/Services/SD
        vfs objects = acl_xattr fruit streams_xattr
        writeable = yes
        browseable = yes
        public = no
        create mode = 0777
        directory mode = 0777
        force create mode = 0777
        force directory mode = 0777
        valid users = root, @SD
        force user = root
        force group = SD
        write list = root, @SD
        veto files = /._*/.DS_Store/*Thumbs.db*/.Trashes/.TemporaryItems/*Thumbs.db*/.Trashes/.TemporaryItems/

...

[michaelfresco]

off-topic: it might be cool to package the tool into a docker image that can be run together with Elastic Search.

[michaelfresco]

@Beleggrodion: I think that you should put the spotlight = yes flag under a share, not under global.

[MatthiasKuehneEllerhold]

Spotlight = yes in [global] will work too.

We have some problems with the spotlight search too. Certain files are never found in their directory. Moving them to another directory suddenly makes them findable. @slow from SerNet is currently on it and has a patch for samba that improves this situation A LOT. Track this bug in the samba bug reporter: https://bugzilla.samba.org/show_bug.cgi?id=15342

[vit9696]

The merge request is not linked by the patch, but it is hanging here: https://gitlab.com/samba-team/samba/-/merge_requests/3056. There are a few more improvements besides the patch in the bugzilla.

Just in case, I have not yet tried building this myself, but at the very least it looks promising.

[MatthiasKuehneEllerhold]

I managed to build a package and deploy it. In our test case the search now works - without the patch certain search queries wont return anything.

So this fix works for us. Ive communicated that to Sernet and Im hoping this patch lands in the main repo soon.

[vit9696]

Glad to hear that. Might you know why the patch is still marked draft at the moment, by the way?

[vit9696]

The changes were merged and apparently will be part of samba 4.17.11, 4.18.6, and 4.19.0.

[vit9696]

So, all these versions with Spotlight search fixes were released. For now only Debian 12 packages 4.18.6 in bookworm-backports. Other Debian releases and repositories package older releases.

I upgraded and tried performing search, however, running mdfind on macOS:

mdfind -onlyin /Volumes/Share Term

Produces no search output. In the logs I see:

[2023/09/07 14:56:58,  0] ../../source3/rpc_server/mdssvc/mdssvc_es.c:755(mds_es_search_http_read_done)
  mds_es_search_http_read_done: json_loadb failed

The issue seems to be known but is separate from this one. I can find the following in the mailing list: https://lists.samba.org/archive/samba/2023-August/246274.html @MatthiasKuehneEllerhold, I guess the message is yours :)

[vit9696]

Maybe it is not quite the right place to ask, but has anyone considered trying API-compatible alternatives to elasticsearch? For our relatively simple use cases perhaps they can be less resource consuming as well.

Quick googling led me to https://github.com/zincsearch/zincsearch, but it is hard to estimate how ready it is to serve Samba, and, no less importantly, interact with fs2es-indexer.

[MatthiasKuehneEllerhold]

@Beleggrodion Is this issue still present? Did you check that your samba version is up to date (look at https://github.com/Ellerhold/fs2es-indexer/issues/27#issuecomment-1676052344)? Make sure not to upgrade your ElasticSearch past 8.8.x (see https://github.com/Ellerhold/fs2es-indexer/issues/27#issuecomment-1710032462).

[Beleggrodion]

I don't have much time and the server is currently running productive. Currently the newest samba for ubuntu 20.04 lts is installed, but this is 4.15.13 . A update to a newer ubuntu is currently not possible. And also ubuntu 22.04 lts has 4.15.13 . Newer samba is is only in the newest non LTS version available.