Closed michlschmid closed 6 years ago
dothebart
commented
7 years ago Hi,
can you re-attempt the installation and have a look at /var/log/arangodb3/arangod.log and /var/log/arangodb3/arangod.log.supervisor and see whether there are any additional messages?
fingolfin
commented
7 years ago I also just encountered this error.
In my case, it seems that somehow the upgrade lead to a corrupted DB. My log now ends like this (no corruption before the upgrade attempt was shown in the log files)
2017-11-09T15:00:06Z [12138] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T15:00:06Z [12138] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T15:00:06Z [12138] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-09T15:00:06Z [12138] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-09T15:00:06Z [12138] INFO using storage engine mmfiles
2017-11-09T15:00:06Z [12138] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-09T15:00:06Z [12138] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-09T15:00:06Z [12138] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-09T15:00:06Z [12138] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
2017-11-09T15:00:06Z [12138] FATAL could not inspect WAL logfiles: corrupted datafile
2017-11-09T15:02:41Z [16639] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T15:02:41Z [16639] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
[ ... a few more similar entries ... ]
2017-11-09T15:19:20Z [1673] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-09T15:19:20Z [1673] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-09T15:19:20Z [1673] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
2017-11-09T15:19:20Z [1673] FATAL could not inspect WAL logfiles: corrupted datafile
2017-11-09T15:22:00Z [1875] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T15:22:00Z [1875] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T15:22:00Z [1875] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-09T15:22:00Z [1875] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-09T15:22:00Z [1875] INFO using storage engine mmfiles
2017-11-09T15:22:00Z [1875] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-09T15:22:00Z [1875] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-09T15:22:01Z [1875] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-09T15:22:01Z [1875] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
2017-11-09T15:22:01Z [1875] FATAL could not inspect WAL logfiles: corrupted datafile
``
The previous log contains no indication about any corruption.
Luckily, this is a test system with no important data, so I can probably just wipe it.
michlschmid
commented
7 years ago @dothebart:
root@bser-ardb01:/var/lib/arangodb3-apps/_db/orgraph/orgraph# tail -f /var/log/arangodb3/arangod.log
2017-11-09T16:25:20Z [5735] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T16:25:20Z [5735] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T16:25:20Z [5735] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-09T16:25:20Z [5735] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-09T16:25:20Z [5735] INFO using storage engine mmfiles
2017-11-09T16:25:20Z [5735] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-09T16:25:20Z [5735] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-09T16:25:22Z [5735] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-27462731.db'
2017-11-09T16:25:22Z [5735] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-27462731.db': corrupted datafile
2017-11-09T16:25:22Z [5735] FATAL could not inspect WAL logfiles: corrupted datafileroot@bser-ardb01:/var/lib/arangodb3-apps/_db/orgraph/orgraph# tail -f /var/log/arangodb3/arangod.log.supervisor
2017-10-08T04:25:01Z [13764] INFO {startup} received SIGHUP for supervisor; commanding client [13765] to logrotate.
2017-10-12T06:51:17Z [13764] INFO {startup} received SIGINT for supervisor; commanding client [13765] to shut down.
2017-10-12T06:51:22Z [13764] INFO {startup} waitpid woke up with return value 13765 and status 0 and DONE = true
2017-10-12T06:51:42Z [1250] INFO {startup} supervisor has forked a child process with pid 1251
2017-10-16T04:25:02Z [1250] INFO {startup} received SIGHUP for supervisor; commanding client [1251] to logrotate.
2017-10-22T04:25:01Z [1250] INFO {startup} received SIGHUP for supervisor; commanding client [1251] to logrotate.
2017-10-29T05:25:02Z [1250] INFO {startup} received SIGHUP for supervisor; commanding client [1251] to logrotate.
2017-11-05T05:25:02Z [1250] INFO {startup} received SIGHUP for supervisor; commanding client [1251] to logrotate.
2017-11-09T07:33:31Z [1250] INFO {startup} received SIGINT for supervisor; commanding client [1251] to shut down.
2017-11-09T07:33:36Z [1250] INFO {startup} waitpid woke up with return value 1251 and status 0 and DONE = true@fingolfin:
dothebart
commented
7 years ago Hi,
If you inspect /var/lib/arangodb - do you see whether the owner of these datafiles is set properly?
If you started ArangoDB as root, it may not be readable for the system user wanting to start the daemon.
fingolfin
commented
7 years ago I assume you mean /var/lib/arangodb3. Well, there are indeed a few files whose group is "root", but for all, the owner is "arangodb. I "chown"'d them, but no visible difference. So I changed thearangodb3.postinstscript by addingset -xat the top. This showed me that the error comes from this command:/usr/sbin/arangod --uid arangodb --gid arangodb --server.rest-server false --log.foreground-tty false --database.check-version`
So I tried this:
# /usr/sbin/arangod --uid arangodb --gid arangodb --server.rest-server false --database.check-version
2017-11-09T16:43:22Z [3622] WARNING {memory} environment variable GLIBCXX_FORCE_NEW' is not set. it is recommended to set it to some value to avoid unnecessary memory pooling in glibc++
2017-11-09T16:43:22Z [3622] WARNING {memory} execute 'export GLIBCXX_FORCE_NEW=1'
2017-11-09T16:43:22Z [3622] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T16:43:22Z [3622] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-09T16:43:22Z [3622] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-09T16:43:22Z [3622] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-09T16:43:22Z [3622] INFO using storage engine mmfiles
2017-11-09T16:43:22Z [3622] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-09T16:43:22Z [3622] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-09T16:43:22Z [3622] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-09T16:43:22Z [3622] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
2017-11-09T16:43:22Z [3622] FATAL could not inspect WAL logfiles: corrupted datafile
#
# ls -al /var/lib/arangodb3/journals/logfile-5867495.db
-rw-rw---- 1 arangodb arangodb 33554432 Nov 9 16:00 /var/lib/arangodb3/journals/logfile-5867495.dbHi,
please keep a copy of /var/lib/arangodb3 (yes 3 actually ;-)) and try whether running
/usr/sbin/arangod --wal.ignore-logfile-errors true
or even more forcefull
/usr/sbin/arangod --wal.ignore-logfile-errors true --wal.ignore-recovery-errors true
can solve your situation.
Which filesystems do you use?
michlschmid
commented
7 years ago @dothebart following rights in my folders:
root@bser-ardb01:/var/lib# ls -la
insgesamt 172
drwxr-xr-x 44 root root 4096 Aug 21 15:20 .
drwxr-xr-x 14 root root 4096 Aug 21 15:20 ..
drwxr-xr-x 4 root root 4096 Mär 6 2017 AccountsService
drwxr-xr-x 3 root root 4096 Mär 6 2017 apparmor
drwxr-xr-x 6 root root 4096 Nov 9 17:25 apt
drwxr-xr-x 5 arangodb arangodb 4096 Nov 9 17:25 arangodb3
drwxr-xr-x 3 arangodb arangodb 4096 Mär 6 2017 arangodb3-apps
root@bser-ardb01:/var/lib/arangodb3# ll
insgesamt 36
drwxr-xr-x 5 arangodb arangodb 4096 Nov 9 17:25 ./
drwxr-xr-x 44 root root 4096 Aug 21 15:20 ../
drwxr-xr-x 4 arangodb arangodb 4096 Mär 6 2017 databases/
-rw-r----- 1 arangodb arangodb 7 Jul 25 09:18 ENGINE
drwxr-xr-x 2 arangodb arangodb 4096 Nov 9 08:33 journals/
-rw------- 1 arangodb arangodb 4 Nov 9 17:25 LOCK
drwxr-xr-x 2 arangodb arangodb 4096 Nov 9 17:25 rocksdb/
-rw-r----- 1 arangodb arangodb 67 Mär 6 2017 SERVER
-rw-rw---- 1 arangodb arangodb 137 Nov 9 08:33 SHUTDOWNAll database and journal folders an its contents seem to be assigned to arangodb:arangodb.
michlschmid
commented
7 years ago My filesystem is ext4 .
fingolfin
commented
7 years ago @vlabmichl sorry for "hijacking" your issue. Perhaps you could also tell the content (or end of content) of your /var/log/arangodb3/arangod.log to see if our two problems are related after all or not?
fingolfin
commented
7 years ago @dothebart I tried both of your suggested commands, in slightly altered form:
[ root ] /var/lib# /usr/sbin/arangod --uid arangodb --gid arangodb --server.rest-server false --database.check-version --wal.ignore-logfile-errors true
2017-11-10T09:11:30Z [4290] WARNING {memory} environment variable GLIBCXX_FORCE_NEW' is not set. it is recommended to set it to some value to avoid unnecessary memory pooling in glibc++
2017-11-10T09:11:30Z [4290] WARNING {memory} execute 'export GLIBCXX_FORCE_NEW=1'
2017-11-10T09:11:30Z [4290] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T09:11:30Z [4290] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T09:11:30Z [4290] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-10T09:11:30Z [4290] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-10T09:11:30Z [4290] INFO using storage engine mmfiles
2017-11-10T09:11:30Z [4290] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-10T09:11:30Z [4290] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-10T09:11:30Z [4290] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-10T09:11:30Z [4290] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
[ root ] /var/lib#and then
[ root ] /var/lib# /usr/sbin/arangod --uid arangodb --gid arangodb --server.rest-server false --database.check-version --wal.ignore-logfile-errors true --wal.ignore-recovery-errors true
2017-11-10T09:11:49Z [4397] WARNING {memory} environment variable GLIBCXX_FORCE_NEW' is not set. it is recommended to set it to some value to avoid unnecessary memory pooling in glibc++
2017-11-10T09:11:49Z [4397] WARNING {memory} execute 'export GLIBCXX_FORCE_NEW=1'
2017-11-10T09:11:49Z [4397] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T09:11:49Z [4397] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T09:11:49Z [4397] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-10T09:11:49Z [4397] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-10T09:11:49Z [4397] INFO using storage engine mmfiles
2017-11-10T09:11:49Z [4397] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-10T09:11:49Z [4397] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-10T09:11:49Z [4397] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-10T09:11:49Z [4397] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafileI verified after each command that no arangodbd process (or any process involving the string arango) was running. For reference, the relevant file is read/write for user:group arangodb:arangodb:
[ root ] /var/lib# ls -al /var/lib/arangodb3/journals/logfile-5867495.db
-rw-rw---- 1 arangodb arangodb 33554432 Nov 9 16:00 /var/lib/arangodb3/journals/logfile-5867495.dbOh, and I am also using ext4 (and ubuntu/single server/etc., i.e. more or less the same setup @vlabmichl seems to be using, with essentially the same things happening, which is why I commented here instead of opening a new issue. But if that's causing too much confusion, I can move over to a new issue)
dothebart
commented
7 years ago Ok, in the community slack we also had somebody who couldn't install arangodb in first place due to the libstdc++ referenced could not be installed:
before the security update:
dpkg -l libstdc++6
ii libstdc++6:amd64 5.4.0-6ubuntu1~16.04.4 amd64 GNU Standard C++ Library v3(which our build containers also use)
After the upgrade:
ii libstdc++6:amd64 5.4.0-6ubuntu1~16.04.5 amd64 GNU Standard C++ Library v3and here the package won't install in first place.
Since the both of you did upgrades, can you confirm you've installed the later libstdc++ on your system?
fingolfin
commented
7 years ago Over here, its this:
ii libstdc++6:amd64 5.4.0-6ubuntu1~16.04.5 amd64 GNU Standard C++ Library v3
ii libstdc++6:i386 5.4.0-6ubuntu1~16.04.5 i386 GNU Standard C++ Library v3Ok, so while it refuses to install on such systems in first place, it will upgrade arangodb / libstdc++.
Can you give us a copy of your /var/lib/arangodb3/ ? I will also try to create a package with the newer libstdc++ bindings.
ObiWahn
commented
7 years ago I think we should require a fixed libstdc++ version in the future to prevent this from happening. We will get mails by annoyed users who can not upgrade their systems but it is probably better than a failing database.
dothebart
commented
7 years ago @fingolfin @vlabmichl can you retry with http://master.jenkins.arangodb.info/arangodb3-3.2.6-1_amd64.deb ? That was built with the newer gcc from the ubuntu security updates.
fingolfin
commented
7 years ago Don't see a difference:
[ root ] /tmp# curl -O http://master.jenkins.arangodb.info/arangodb3-3.2.6-1_amd64.deb
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 33.4M 100 33.4M 0 0 9944k 0 0:00:03 0:00:03 --:--:-- 9945k
[ root ] /tmp# dpkg -i arangodb3-3.2.6-1_amd64.deb
(Reading database ... 122780 files and directories currently installed.)
Preparing to unpack arangodb3-3.2.6-1_amd64.deb ...
Unpacking arangodb3 (3.2.6) over (3.2.6) ...
Setting up arangodb3 (3.2.6) ...
FATAL ERROR: EXIT_FAILED - "exit with error"
dpkg: error processing package arangodb3 (--install):
subprocess installed post-installation script returned error exit status 1
Processing triggers for systemd (229-4ubuntu21) ...
Processing triggers for ureadahead (0.100.0-19) ...
ureadahead will be reprofiled on next reboot
Processing triggers for man-db (2.7.5-1) ...
Errors were encountered while processing:
arangodb3
[ root ] /tmp# /usr/sbin/arangod --uid arangodb --gid arangodb --server.rest-server false --database.check-version
2017-11-10T14:24:44Z [5304] WARNING {memory} environment variable GLIBCXX_FORCE_NEW' is not set. it is recommended to set it to some value to avoid unnecessary memory pooling in glibc++
2017-11-10T14:24:44Z [5304] WARNING {memory} execute 'export GLIBCXX_FORCE_NEW=1'
2017-11-10T14:24:44Z [5304] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T14:24:44Z [5304] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-10T14:24:44Z [5304] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-10T14:24:44Z [5304] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-10T14:24:44Z [5304] INFO using storage engine mmfiles
2017-11-10T14:24:44Z [5304] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-10T14:24:44Z [5304] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-10T14:24:44Z [5304] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-5867495.db'
2017-11-10T14:24:44Z [5304] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-5867495.db': corrupted datafile
2017-11-10T14:24:44Z [5304] FATAL could not inspect WAL logfiles: corrupted datafileWill
/usr/sbin/arangod --wal.ignore-logfile-errors true --wal.ignore-recovery-errors true
be doing any good for you?
Maybe from the first backup of the database directory?
dothebart
commented
7 years ago @vlabmichl @fingolfin can you send us a copy of your /var/lib/arangodb3/ via i.e. dropbox so we can look into the actual problem?
If you need an NDA signed for this please let us know.
You may reach us via hackers [at) arangodb dot com , please reference this ticket.
clirix
commented
7 years ago I have this problem too. It occurs after an "apt-get upgrade". The system is an Ubuntu LTS 16.04
2017-11-16T10:42:25Z [6295] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2017-11-16T10:42:25Z [6295] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2017-11-16T10:42:25Z [6295] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2017-11-16T10:42:25Z [6295] INFO using storage engine mmfiles
2017-11-16T10:42:25Z [6295] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-11-16T10:42:25Z [6295] INFO Authentication is turned on (system only), authentication for unix sockets is turned on
2017-11-16T10:42:25Z [6295] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-20059012.db'
2017-11-16T10:42:25Z [6295] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-20059012.db': corrupted datafile
2017-11-16T10:42:25Z [6295] FATAL could not inspect WAL logfiles: corrupted datafileCan you please upgrade to arangodb 3.2.8 ? we the cause for the corruption should have been fixed.
fingolfin
commented
7 years ago I already moved to a new DB, so I don't really want to re-install the corrupt files to test that.
@dothebart I emailed you a link to the corrupt files right after you requested it, did you get that? Never got a confirmation reply.
dothebart
commented
7 years ago Thanks a lot for taking the time, meanwhile another user already sent files of a similar incident via another channel, so this got lost during my vacation.
However you should really upgrade to 3.2.8 to stop this from happening again.
suciokhan
commented
6 years ago Having same issue with 3.2.9 when I try to run sudo-apt-get upgrade:
After this operation, 0 B of additional disk space will be used. Do you want to continue? [Y/n] y Setting up arangodb3 (3.2.9) ... FATAL ERROR: EXIT_FAILED - "exit with error" dpkg: error processing package arangodb3 (--configure): subprocess installed post-installation script returned error exit status 1 Errors were encountered while processing: arangodb3 E: Sub-process /usr/bin/dpkg returned an error code (1)
dothebart
commented
6 years ago Hi, did you try to clean up your database as stated in https://github.com/arangodb/arangodb/issues/3632#issuecomment-343220495 ?
suciokhan
commented
6 years ago I copied /var/lib/arangodb3 to /var/lib/arangodb3.backup.
Then tried both commands: /usr/sbin/arangod --wal.ignore-logfile-errors true /usr/sbin/arangod --wal.ignore-logfile-errors true --wal.ignore-recovery-errors true
Same results both times: 2017-12-07T13:54:04Z [3376] INFO ArangoDB 3.2.9 [linux] 64bit, using jemalloc, VPack 0.1.30, RocksDB 5.6.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.0.2g 1 Mar 2016 2017-12-07T13:54:04Z [3376] WARNING {memory} environment variable GLIBCXX_FORCE_NEW' is not set. it is recommended to set it to some value to avoid unnecessary memory pooling in glibc++ 2017-12-07T13:54:04Z [3376] WARNING {memory} execute 'export GLIBCXX_FORCE_NEW=1' 2017-12-07T13:54:04Z [3376] WARNING {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise' 2017-12-07T13:54:04Z [3376] WARNING {memory} /sys/kernel/mm/transparent_hugepage/defrag is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise' 2017-12-07T13:54:04Z [3376] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"' 2017-12-07T13:54:04Z [3376] WARNING {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"' 2017-12-07T13:54:04Z [3376] INFO using storage engine mmfiles 2017-12-07T13:54:04Z [3376] INFO {cluster} Starting up with role SINGLE 2017-12-07T13:54:04Z [3376] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576 2017-12-07T13:54:04Z [3376] INFO Authentication is turned on (system only), authentication for unix sockets is turned on 2017-12-07T13:54:04Z [3376] ERROR corrupted datafile header read from '/var/lib/arangodb3/journals/logfile-4914328.db' 2017-12-07T13:54:04Z [3376] ERROR unable to open logfile '/var/lib/arangodb3/journals/logfile-4914328.db': corrupted datafile 2017-12-07T13:54:04Z [3376] INFO using endpoint 'http+tcp://127.0.0.1:8529' for non-encrypted requests 2017-12-07T13:54:07Z [3376] INFO ArangoDB (version 3.2.9 [linux]) is ready for business. Have fun!
However, when I run sudo apt-get upgrade I get:
Setting up arangodb3 (3.2.9) ... FATAL ERROR: EXIT_COULD_NOT_LOCK - "could not lock - another process could be running" dpkg: error processing package arangodb3 (--configure): subprocess installed post-installation script returned error exit status 22 Errors were encountered while processing: arangodb3 E: Sub-process /usr/bin/dpkg returned an error code (1)
dothebart
commented
6 years ago Once you got the arangod up and runnig (temporarily) you should dump the database. Then you stop it, install with a fresh database directory, an restore.
The last error you've posted indicates that there is an arangodb process running in a phase its expected to be gone.
suciokhan
commented
6 years ago For reinstall, I'm utilizing your guidance here: https://stackoverflow.com/questions/38425954/error-upgrading-arangodb-3-0-3-from-apt/38428843
rm -f /var/cache/apt/archives/arango* apt-get --reinstall install arangodb3
However, I receive this message:
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
E: Internal Error, No file name for arangodb3:amd64
dothebart
commented
6 years ago hm, did you dpkg --purge arangodb3 in advance? (keep a backup of your /var/lib/arangodb and if you've modified any more /var/lib/arangodb3-apps and /etc/arangodb3)
suciokhan
commented
6 years ago Thanks very much.
I can now run sudo apt-get upgrade with no errors.
Tamal
commented
6 years ago Upgrading from 3.2.8 to 3.2.9 also facing this issue.
dimanjy
commented
6 years ago Same error on 3.2.9 - "corrupted datafile" on autostart this morning. Yesterday everything was fine. I removed corrupted files, made upgrade to ArangoDB 3.2.9 enterprise and restarted. And it's OK now.
ObiWahn
commented
6 years ago Hi all,
in pre 3.2.8 releases we had trouble with data corruption. This should be fixed by now. Do you use datafiles that originate in versions prior to 3.2.8?
dimanjy
commented
6 years ago I start lerning ArangoDB from version 3.2.8 at 2017-11-20
ObiWahn
commented
6 years ago @dimanjy could you provide the datafile to us? If not then it would be good if you could look into the file and find out if there are any log messages in the data.
dimanjy
commented
6 years ago Something's wrong with GinHub file attachment... Where should I send the corrupted file? E-mail or upload somewhere?
ObiWahn
commented
6 years ago Please send it to - hackers at arangodb.com - and put the issue number in the topic. Thank you very much!
dimanjy
commented
6 years ago logfile-3675835.db.gz I did it! (access rights issue :) This file was mentioned as corrupted in my ArangoDb logs.
ObiWahn
commented
6 years ago The first line of the log contains: 2017-12-17T21:54:39Z [2581] INFO control-c received, beginning shut down sequence We are investigating! What OS do you use and how is your arangodb started? Do you use systemd? Is the systemd unit auto-generated or not?
dimanjy
commented
6 years ago I'm on Ubuntu 14.04 and I installed ArangoDb from https://www.arangodb.com/download-major/ubuntu/ (Ubuntu 14.04 section - Installation via Package Manager) I'm not sure what startup system I'm using... I guess that it's Ubuntu default startup system. I guess it's Upstart.
ObiWahn
commented
6 years ago https://github.com/arangodb/arangodb/pull/4116 - this should be the fix - It will be included in th upcoming 3.3.1
sleto-it
commented
6 years ago 3.3.1 has been released and it includes the fix corresponding to the one cited in previous comment by @ObiWahn (PR https://github.com/arangodb/arangodb/pull/4115)
Please let us know if you still see this problem. Many thanks,
my environment running ArangoDB
I'm using the latest ArangoDB of the respective release series:
Mode:
On this operating system:
this is an installation-related issue:
I got the following error while trying to upgrade Arangodb 3.2.4 to 3.2.6 with apt on my ubuntu server:
I tried uninstalling -> reinstalling arangodb completly with no success:
Also trying to re-install 3.2.4 failed with the same error. Before each install command I forced apt to clean its package cache and update sources:
Thanks for any help.