search

grosjo / fts-xapian

Dovecot FTS plugin based on Xapian
GNU Lesser General Public License v2.1
97 stars 21 forks source link

Build issue on FreeBSD #101

Closed jurajlutter closed 2 years ago

jurajlutter commented 3 years ago

Hi,

while building 1.4.14 on FreeBSD, I'm getting:

--- fts-backend-xapian.lo ---
In file included from fts-backend-xapian.cpp:53:
./fts-backend-xapian-functions.cpp:508:22: error: use of undeclared identifier '_SC_AVPHYS_PAGES'
        return long(sysconf(_SC_AVPHYS_PAGES)*sysconf(_SC_PAGE_SIZE) / 1024.0);
                            ^

_SC_AVPHYS_PAGES is nonportable construct, Linux specific. On FreeBSD, this value needs to be computed in a different manner. I'm happy to find out the proper way.

grosjo commented 3 years ago

Yes, if you have a simple way to get the free memory, in a FreeBSD manner, do not hesitate to push a PR

grosjo commented 3 years ago

I have made some updates

Can you check it ?

grosjo commented 3 years ago

@jurajlutter please kindly test with latest git

jurajlutter commented 3 years ago

Builds fine.

grosjo commented 3 years ago

if you put "verbose=2", you shell see in the log the "free memory" calculated. Can you check it is consistent with reality ?

pmezhuev commented 3 years ago 
FTS Xapian: Index path = /var/mail/mdbox/mpa@example.com/xapian-indexes
FTS Xapian: Starting with partial=3 full=20 verbose=2 lowmemory=200
FTS Xapian: fts_backend_xapian_get_last_uid
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Index path = /var/mail/mdbox/mpa@example.com/xapian-indexes
FTS Xapian: 'Drafts' (/var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1) indexes do not exist. Initializing DB
FTS Xapian: fts_backend_xapian_open_readonly
FTS Xapian: Opening DB (RO) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Get last UID of Drafts (88518f0504aa4e599485000046f086e1) = 0
FTS Xapian: fts_backend_xapian_get_last_uid
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Box is unchanged
FTS Xapian: fts_backend_xapian_open_readonly
FTS Xapian: Opening DB (RO) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Get last UID of Drafts (88518f0504aa4e599485000046f086e1) = 0
FTS Xapian: fts_backend_update_context
FTS Xapian: fts_backend_xapian_update_set_mailbox
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Box is unchanged
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: fts_backend_xapian_oldbox
FTS Xapian: Start indexing 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: New part (Header=Message-ID,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: Opening DB (RW) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Opening DB (RW) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1 (0 docs stored): Done
Free memory 137442247796
Warning: FTS Xapian: Free memory 134220945 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(XMID) -> 3 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Date,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'date' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=MIME-Version,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'mimeversion' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=User-Agent,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'useragent' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Language,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contentlanguage' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=From,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
Free memory 3291300
Warning: FTS Xapian: Free memory 3214 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(A) -> 83 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Subject,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
Free memory 137442244772
Warning: FTS Xapian: Free memory 134220942 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(S) -> 3 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Organization,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'organization' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=X-Mozilla-Draft-Info,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'xmozilladraftinfo' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=X-Identity-Key,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'xidentitykey' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Fcc,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'fcc' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Type,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttype' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Transfer-Encoding,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttransferencoding' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=(null),Type=text/html,Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
Free memory 11303151335520420
Warning: FTS Xapian: Free memory 11038233726094 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_text
FTS Xapian: fts_backend_xapian_query
FTS Xapian: NGRAM(body,XBDY) -> 448 items, max length=32, (total 5 KB)
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_mailbox
FTS Xapian: Unset box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: fts_backend_xapian_oldbox
FTS Xapian: Done indexing 'Drafts' (88518f0504aa4e599485000046f086e1) (1 msgs in 47 ms, rate: 21.3)
FTS Xapian: fts_backend_xapian_release (unset_box)
FTS Xapian: Committed 'unset_box' in 1 ms (1 docs in index)
FTS Xapian: Box is empty
FTS Xapian: fts_backend_xapian_update_deinit (/var/mail/mdbox/mpa@example.com/xapian-indexes)
FTS Xapian: fts_backend_xapian_release (update_deinit)
FTS Xapian: Committed 'update_deinit' in 0 ms
FTS Xapian: Deinit /var/mail/mdbox/mpa@example.com/xapian-indexes)

top at this time:

Mem: 2202M Active, 10G Inact, 1009M Laundry, 15G Wired, 3193M Free
Swap: 16G Total, 85M Used, 16G Free

3214 MB match free-type memory. 134220945 MB looks strange.

jurajlutter commented 3 years ago

This code will get you the amount of free memory, in pages (hw.pagesize)

#include <sys/types.h>
#include <sys/sysctl.h>
#include <stdio.h>

int main(int argc, char **argv)
{
        size_t len;
        uint32_t m;
        len = sizeof(m);
        sysctlbyname("vm.stats.vm.v_free_count", &m, &len, NULL, 0);
        printf("free %d pages\n", m);
}

m should be uint32_t.

grosjo commented 3 years ago

@jurajlutter thanks @LordBeaver Can you test again ?

pmezhuev commented 3 years ago 
FTS Xapian: Index path = /var/mail/mdbox/mpa@example.com/xapian-indexes
FTS Xapian: Starting with partial=3 full=20 verbose=2 lowmemory=200
FTS Xapian: fts_backend_xapian_get_last_uid
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Index path = /var/mail/mdbox/mpa@example.com/xapian-indexes
FTS Xapian: 'Drafts' (/var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1) indexes do not exist. Initializing DB
FTS Xapian: fts_backend_xapian_open_readonly
FTS Xapian: Opening DB (RO) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Get last UID of Drafts (88518f0504aa4e599485000046f086e1) = 0
FTS Xapian: fts_backend_xapian_get_last_uid
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Box is unchanged
FTS Xapian: fts_backend_xapian_open_readonly
FTS Xapian: Opening DB (RO) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Get last UID of Drafts (88518f0504aa4e599485000046f086e1) = 0
FTS Xapian: fts_backend_update_context
FTS Xapian: fts_backend_xapian_update_set_mailbox
FTS Xapian: Set box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: Box is unchanged
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: fts_backend_xapian_oldbox
FTS Xapian: Start indexing 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: New part (Header=MIME-Version,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'mimeversion' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Date,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'date' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=From,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: Opening DB (RW) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1
FTS Xapian: Opening DB (RW) /var/mail/mdbox/mpa@example.com/xapian-indexes/db_88518f0504aa4e599485000046f086e1 (0 docs stored): Done
FTS Xapian: (BSD) Free pages 757391
FTS Xapian: (BSD) Free memory 3029564 kB
Warning: FTS Xapian: Free memory 2958 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(A) -> 83 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=To,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757202
FTS Xapian: (BSD) Free memory 3028808 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(XTO) -> 208 items (total 2 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Subject,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757139
FTS Xapian: (BSD) Free memory 3028556 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(S) -> 3 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Message-ID,Type=(null),Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757139
FTS Xapian: (BSD) Free memory 3028556 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_hdr
FTS Xapian: fts_backend_xapian_query
FTS Xapian: Ngram(XMID) -> 3 items (total 0 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=X-Sender,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'xsender' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Organization,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'organization' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Type,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttype' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Transfer-Encoding,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttransferencoding' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Type,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttype' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=(null),Type=text/plain,Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757139
FTS Xapian: (BSD) Free memory 3028556 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_text
FTS Xapian: fts_backend_xapian_query
FTS Xapian: NGRAM(body,XBDY) -> 431 items, max length=32, (total 5 KB)
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Transfer-Encoding,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttransferencoding' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=Content-Type,Type=(null),Disposition=(null))
FTS Xapian: Unknown header 'contenttype' of part
FTS Xapian: fts_backend_xapian_update_set_build_key
FTS Xapian: New part (Header=(null),Type=text/html,Disposition=(null))
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757139
FTS Xapian: (BSD) Free memory 3028556 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_text
FTS Xapian: fts_backend_xapian_query
FTS Xapian: NGRAM(body,XBDY) -> 17 items, max length=6, (total 0 KB)
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_check_access
FTS Xapian: (BSD) Free pages 757139
FTS Xapian: (BSD) Free memory 3028556 kB
Warning: FTS Xapian: Free memory 2957 MB vs 200 MB minimum
FTS Xapian: fts_backend_xapian_index_text
FTS Xapian: fts_backend_xapian_query
FTS Xapian: NGRAM(body,XBDY) -> 448 items, max length=32, (total 5 KB)
FTS Xapian: Indexing part as text
FTS Xapian: fts_backend_xapian_update_unset_build_key with 1 docs in the index
FTS Xapian: fts_backend_xapian_update_set_mailbox
FTS Xapian: Unset box 'Drafts' (88518f0504aa4e599485000046f086e1)
FTS Xapian: fts_backend_xapian_oldbox
FTS Xapian: Done indexing 'Drafts' (88518f0504aa4e599485000046f086e1) (1 msgs in 50 ms, rate: 20.0)
FTS Xapian: fts_backend_xapian_release (unset_box)
FTS Xapian: Committed 'unset_box' in 1 ms (1 docs in index)
FTS Xapian: Box is empty
FTS Xapian: fts_backend_xapian_update_deinit (/var/mail/mdbox/mpa@example.com/xapian-indexes)
FTS Xapian: fts_backend_xapian_release (update_deinit)
FTS Xapian: Committed 'update_deinit' in 0 ms
FTS Xapian: Deinit /var/mail/mdbox/mpa@example.com/xapian-indexes)
grosjo commented 3 years ago

and 2957 MB is actually the free memory on your system ?

pmezhuev commented 3 years ago

@grosjo, yes.

kour1er commented 2 years ago

Trying to build on macOS 12.1 (intel) here and I'm seeing the same error message on the most recent git committed version:

In file included from fts-backend-xapian.cpp:53: ./fts-backend-xapian-functions.cpp:530:24: error: use of undeclared identifier '_SC_AVPHYS_PAGES' long m = long(sysconf(_SC_AVPHYS_PAGES) * fts_xapian_settings.pagesize / 1024.0); ^

As macOS is BSD based I wondered whether their might be a similar solution? Any suggestions on what to try gratefully received.

grosjo commented 2 years ago

@kour1er Can you try with latest git ?

kour1er commented 2 years ago

@grosjo Thanks for looking into this so quickly! Well just tried with the latest git and I'm getting a different error message now so progress :) I'm seeing

`./fts-xapian-plugin.h:21:10: fatal error: 'vm/vm_param.h' file not found

include <vm/vm_param.h>`

which looks like it's referencing the #if defined that you added for APPLE along with BSD. Not sure if it's something I've done incorrectly in my setup of if the vm/vm_param.h doesn't have an equivalent on macOS. Any thoughts on that or should I just try figuring it out some more!

jurajlutter commented 2 years ago

I might want to have a look how to do this properly on macOS.

grosjo commented 2 years ago

@jurajlutter Any news on that ? Happy new year

jurajlutter commented 2 years ago

On macOS, vm/vm_param.h does not need to be included.

jurajlutter commented 2 years ago

This is the diff I'm using right now:

otis@chemex ~/work/fts-xapian % git diff
diff --git a/src/Makefile.am b/src/Makefile.am
index 5963e7c..2e468ab 100644
--- a/src/Makefile.am
+++ b/src/Makefile.am
@@ -12,7 +12,7 @@ AM_CXXFLAGS = -std=gnu++11 -O2 \
        $(AM_CPPFLAGS) \
        $(XAPIAN_LIBS) \
        $(ICU_LIBS) \
-       -Wl,-rpath=$(ICU_LDFLAGS)
+       -Wl,-rpath -Wl,$(ICU_LDFLAGS)

 lib21_fts_xapian_plugin_la_LDFLAGS = -module -avoid-version

diff --git a/src/fts-backend-xapian.cpp b/src/fts-backend-xapian.cpp
index d93a7a7..0e133a8 100644
--- a/src/fts-backend-xapian.cpp
+++ b/src/fts-backend-xapian.cpp
@@ -706,5 +706,5 @@ struct fts_backend fts_backend_xapian =
                NULL
        },
        .ns = NULL,
-       .updating = NULL
+       .updating = 0
 };
diff --git a/src/fts-xapian-plugin.h b/src/fts-xapian-plugin.h
index d04dc8e..2ae5462 100644
--- a/src/fts-xapian-plugin.h
+++ b/src/fts-xapian-plugin.h
@@ -18,8 +18,10 @@
 #if defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__APPLE__)
 #include <sys/types.h>
 #include <sys/sysctl.h>
+#if !defined(__APPLE__)
 #include <vm/vm_param.h>
 #endif
+#endif

 #define XAPIAN_FILE_PREFIX "xapian-indexes"
 #define XAPIAN_TERM_SIZELIMIT 245L
grosjo commented 2 years ago

Hello, Can you try latest git ?

grosjo commented 2 years ago

@jurajlutter good for you ?

grosjo commented 2 years ago

@jurajlutter All good ?

grosjo commented 2 years ago

@jurajlutter compilation is fine ?

jurajlutter commented 2 years ago

Let me check.

jurajlutter commented 2 years ago

Builds just fine.

jurajlutter commented 2 years ago

There is one minor warning left:

--- fts-backend-xapian.lo ---
fts-backend-xapian.cpp:751:2: warning: mixture of designated and non-designated initializers in the same initializer list is a C99 extension [-Wc99-designator]
        .name = "xapian",
        ^~~~~~~~~~~~~~~~
fts-backend-xapian.cpp:753:2: note: first non-designated initializer is here
        {
        ^
1 warning generated.
jurajlutter commented 2 years ago

A patch like:

--- src/fts-backend-xapian.cpp.orig     2022-03-25 21:16:29 UTC
+++ src/fts-backend-xapian.cpp
@@ -750,7 +750,7 @@ struct fts_backend fts_backend_xapian =
 {
        .name = "xapian",
        .flags = FTS_BACKEND_FLAG_BUILD_FULL_WORDS,
-       {
+       .v = {
                fts_backend_xapian_alloc,
                fts_backend_xapian_init,
                fts_backend_xapian_deinit,

could fix it.

grosjo commented 2 years ago

Ok I fixed that

grosjo commented 2 years ago

Thanks!