twright-msft
commented
7 years ago This seems like a volume mounting issue. Can you please share your docker run command and the details about what you are mounting if anything?
Closed thiagoalves closed 3 years ago
twright-msft
commented
7 years ago This seems like a volume mounting issue. Can you please share your docker run command and the details about what you are mounting if anything?
MatrixDai
commented
7 years ago @twright-msft Here is the full log without mount volume.
$ export DOCKER_HOST=10.69.11.100:2376
$ docker version
Client:
Version: 1.13.1
API version: 1.26
Go version: go1.7.5
Git commit: 092cba3
Built: Wed Feb 8 08:47:51 2017
OS/Arch: darwin/amd64
Server:
Version: 17.03.0-ee-1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 9094a76
Built: Wed Mar 1 01:24:21 2017
OS/Arch: linux/amd64
Experimental: false
$ docker run --name matrix-debug --cpu-period=50000 --cpu-quota=200000 -m24G -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -d microsoft/mssql-server-linux
faeed50079be7e580a9eba32dfc1a47b54b23a3f98987942ac0192a0c87b841a
$ docker logs -f matrix-debug
Configuring Microsoft(R) SQL Server(R)...
sqlservr: Debugger.cpp:578: static void Debugger::Print(const void *, unsigned int): Assertion `"A serious error condition has been encountered." == nullptr' failed.
/opt/mssql/bin/generate-core.sh: line 135: lsof: command not found
cat: core.sqlservr.16.temp/log/debug.log: No such file or directory
No journal files were found.
No journal files were found.
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Configuration complete.
This is an evaluation version. There are [144] days left in the evaluation period.
Unexpected signal encountered in host environment. Signal: 5 (Trace/breakpoint trap) at 0x56079e339cf8
Capturing core dump and information for sqlservr (pid 200)...
sqlservr: Debugger.cpp:578: static void Debugger::Print(const void *, unsigned int): Assertion `"A serious error condition has been encountered." == nullptr' failed.
********** PAL FATAL SIGNAL - Aborted (6) **********
SP IP Function
00007f48b21e3000 000056079e35a23e std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > std::__1::operator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(char const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+0x1e0fe
00007f48b21e3040 00007f48b91ec4b0 killpg+0x40
00007f48b21e35d8 00007f48b91ec428 gsignal+0x38
00007f48b21e35e0 00007f48b91ee02a abort+0x16a
00007f48b21e3710 000056079e339bc0 _init+0x37ed8
00007f48b21e38c0 00007f48bac993e0 __restore_rt+0x0
00007f48b21e3e50 000056079e339cf8 _init+0x38010
00007f48b21e3ee0 00007f48bac8f70a start_thread+0xca
00007f48b21e3f80 00007f48b92bd82d clone+0x6d
00007f48b21e3f88 0000000000000000 +0x6d
*******************************************************************
/opt/mssql/bin/generate-core.sh: line 135: lsof: command not found
Capturing dump with gdb
cat: core.sqlservr.200.temp/log/debug.log: No such file or directory
No journal files were found.
No journal files were found.
SQL Log Directory /var/opt/mssql/log
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Packaging core dump and information into compressed files.
Core dump and information will be written to /var/opt/mssql/log/core.sqlservr.03_23_2017_01_12_06.200 and logs in /var/opt/mssql/log/core.sqlservr.03_23_2017_01_12_06.200.logThe docker host is the same one as @thiagoalves
thiagoalves
commented
7 years ago @twright-msft Were you able to look into this issue? We were able to reproduce it on two production servers, and we are not necessarily mounting any external volumes
twright-msft
commented
7 years ago I have a theory... This is the error in the log: lsof: command not found The assumption with the mssql-server-linux container right now is that the sqlservr process is running as root. Is it possible that you are trying to run this container in something like OpenShift that doesnt allow running containers as root by default?
thiagoalves
commented
7 years ago @twright-msft We have upgraded docker to 17.03 in both servers. Container seems to be running fine in one of them, but I still get issues with the other one:
root@docker_linux_4:~# docker run --name matrix-debug --cpu-period=50000 --cpu-quota=200000 -m24G -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' -d -v /var/opt/mssql/:/tmp microsoft/mssql-server-linux
d56cfb9b7d6e873b3e99acfc1091933d1a0ffc4423a04416160d02da7263983c
^[[Aroot@docker_linux_4:~# doclogs -f matrix-debug
Configuring Microsoft(R) SQL Server(R)...
No journal files were found.
No journal files were found.
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.I don't see any useful info in the logs:
root@docker_linux_4:~# docker start matrix-debug; docker exec matrix-debug cat /var/opt/mssql/log/*
matrix-debug
set pagination off
set height 0
set width 0
attach 22
gcore core.sqlservr.22.temp/core.sqlservr.22
set logging overwrite on
set logging file core.sqlservr.22.temp/log/debug.log.gdb
set logging on
echo === Process information ===\n
info inferior 1
echo \n
echo === Shared library information ===\n
info sharedlibrary
echo \n
echo === Thread stacks ===\n
thread apply all bt
echo \n
echo === Thread registers ===\n
thread apply all info all-reg
echo \n
set logging off
kill inferior 1
quit
cat: /var/opt/mssql/log/core.sqlservr.03_30_2017_13_21_22.16.log: No such file or directory
cat: /var/opt/mssql/log/core.sqlservr.16.temp: Is a directory
cat: /var/opt/mssql/log/setup-20170330-132122.log: No such file or directoryWe are running the container as root user
thiagoalves
commented
7 years ago @twright-msft Here's how I reproduce the issue:
1) get a shell inside the container:
docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' --name matrix-debug -it microsoft/mssql-server-linux bash
2) Run this on the container:
/opt/mssql/bin/sqlservr --help
This is the output:
This is an evaluation version. There are [158] days left in the evaluation period.
Unexpected signal encountered in host environment. Signal: 5 (Trace/breakpoint trap) at 0x55d2b84db558
********** PAL FATAL SIGNAL - Aborted (6) **********
SP IP Function
00007f3b377fd000 000055d2b84ada8e _init+0x19ad6
00007f3b377fd040 00007f3b3f67c4b0 killpg+0x40
00007f3b377fd5d8 00007f3b3f67c428 gsignal+0x38
00007f3b377fd5e0 00007f3b3f67e02a abort+0x16a
00007f3b377fd710 000055d2b84db420 _init+0x47468
00007f3b377fd8c0 00007f3b41a783e0 __restore_rt+0x0
00007f3b377fde50 000055d2b84db558 _init+0x475a0
00007f3b377fdee0 00007f3b41a6e70a start_thread+0xca
00007f3b377fdf80 00007f3b3f74d82d clone+0x6d
00007f3b377fdf88 0000000000000000 +0x6d
*******************************************************************
Dump control values set to default. Behavior controlled with /var/opt/mssql/mssql.conf
===============================================
Mini dump type: miniplus (2)
Capturing mini and full dump: true
===============================================
Capturing core dump and information for sqlservr (pid 12)...
Capturing dump with paldumper
Attempt to capture dump with paldumper failed. Reference core.sqlservr.12.temp/log/debug.log for details
Capturing dump with gdb
No journal files were found.
No journal files were found.
SQL Log Directory /var/opt/mssql/log
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Packaging core dump and information into compressed files.
Core dump and information will be written to /var/opt/mssql/log/core.sqlservr.04_06_2017_13_35_50.12 and logs in /var/opt/mssql/log/core.sqlservr.04_06_2017_13_35_50.12.log
root@4ed0705a72fa:/# nohup: redirecting stderr to stdoutYou will find the dump and logs here:
twright-msft
commented
7 years ago Thanks for the details @thiagoalves . We'll take a look!
twright-msft
commented
7 years ago Looks like we can't take a proper dump because of this error: Could not attach to process. If your uid matches the uid of the target process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf /var/opt/mssql/log/gdbcommands:4: Error in sourced command file: ptrace: Operation not permitted.
More info on this issue: http://askubuntu.com/questions/146160/what-is-the-ptrace-scope-workaround-for-wine-programs-and-are-there-any-risks
You could try running this command on your host before running it again to see if it will yield a proper dump.
echo 0|sudo tee /proc/sys/kernel/yama/ptrace_scope
What distro/version are you using for the host?
thiagoalves
commented
7 years ago @twright-msft The output seems to be the same after the change:
root@docker_linux_4:~# cat /proc/sys/kernel/yama/ptrace_scope
0
root@docker_linux_4:~# docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=yourStrong(!)Password' --name matrix-debug -it microsoft/mssql-server-linux bash
root@a06994b7a4e3:/# /opt/mssql/bin/sqlservr --help
This is an evaluation version. There are [157] days left in the evaluation period.
Unexpected signal encountered in host environment. Signal: 5 (Trace/breakpoint trap) at 0x56242af73558
********** PAL FATAL SIGNAL - Aborted (6) **********
SP IP Function
00007f46c13fd000 000056242af45a8e _init+0x19ad6
00007f46c13fd040 00007f46c93d84b0 killpg+0x40
00007f46c13fd5d8 00007f46c93d8428 gsignal+0x38
00007f46c13fd5e0 00007f46c93da02a abort+0x16a
00007f46c13fd710 000056242af73420 _init+0x47468
00007f46c13fd8c0 00007f46cb7d43e0 __restore_rt+0x0
00007f46c13fde50 000056242af73558 _init+0x475a0
00007f46c13fdee0 00007f46cb7ca70a start_thread+0xca
00007f46c13fdf80 00007f46c94a982d clone+0x6d
00007f46c13fdf88 0000000000000000 +0x6d
*******************************************************************
Dump control values set to default. Behavior controlled with /var/opt/mssql/mssql.conf
===============================================
Mini dump type: miniplus (2)
Capturing mini and full dump: true
===============================================
Capturing core dump and information for sqlservr (pid 13)...
Capturing dump with paldumper
Attempt to capture dump with paldumper failed. Reference core.sqlservr.13.temp/log/debug.log for details
Capturing dump with gdb
No journal files were found.
No journal files were found.
SQL Log Directory /var/opt/mssql/log
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Packaging core dump and information into compressed files.
Core dump and information will be written to /var/opt/mssql/log/core.sqlservr.04_07_2017_12_43_33.13 and logs in /var/opt/mssql/log/core.sqlservr.04_07_2017_12_43_33.13.log
nohup: redirecting stderr to stdoutYou will find the new dump and logs here:
Output of docker info:
Containers: 288
Running: 276
Paused: 0
Stopped: 12
Images: 195
Server Version: 17.03.0-ee-1
Storage Driver: aufs
Root Dir: /opt/io1/docker/aufs
Backing Filesystem: extfs
Dirs: 2273
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-66-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 240.1 GiB
Name: docker_linux_4
ID: MOMS:J3B5:X6GA:TY6P:6LWQ:XCT2:6RAG:5EIV:TCOK:DBXI:CGA4:2SMG
Docker Root Dir: /opt/io1/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 2688
Goroutines: 1732
System Time: 2017-04-07T12:49:57.121035269Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
root@docker_linux_4:~# docker -D info
Containers: 289
Running: 276
Paused: 0
Stopped: 13
Images: 195
Server Version: 17.03.0-ee-1
Storage Driver: aufs
Root Dir: /opt/io1/docker/aufs
Backing Filesystem: extfs
Dirs: 2275
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 977c511eda0925a723debdc94d09459af49d082a
runc version: a01dafd48bc1c7cc12bdb01206f9fea7dd6feb70
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-66-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 240.1 GiB
Name: docker_linux_4
ID: MOMS:J3B5:X6GA:TY6P:6LWQ:XCT2:6RAG:5EIV:TCOK:DBXI:CGA4:2SMG
Docker Root Dir: /opt/io1/docker
Debug Mode (client): true
Debug Mode (server): true
File Descriptors: 2701
Goroutines: 1748
System Time: 2017-04-07T12:50:05.08920553Z
EventsListeners: 3
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: trueOutput of docker version:
root@docker_linux_4:~# docker version
Client:
Version: 17.03.0-ee-1
API version: 1.26
Go version: go1.7.5
Git commit: 9094a76
Built: Wed Mar 1 01:24:21 2017
OS/Arch: linux/amd64
Server:
Version: 17.03.0-ee-1
API version: 1.26 (minimum version 1.12)
Go version: go1.7.5
Git commit: 9094a76
Built: Wed Mar 1 01:24:21 2017
OS/Arch: linux/amd64
Experimental: falseI'm not sure I understand why you are using -it w/ the bash prompt and the running sqlservr. When the container starts up with docker run the CMD will execute sqlservr.sh which will start sqlservr. It shouldnt be necessary to start sqlservr manually after the container is started.
Can you please try to docker run -d and see if sqlservr starts up OK and you can connect to it?
thiagoalves
commented
7 years ago @twright-msft If you take a look at the previous messages, we were always running the container with -d previously. The containers starts and stops itself after a few seconds.
That's why I tried -it - to keep the container running and run sqlservr.sh manually and try to troubleshoot the issue. The script failed on line 90:
SQLSERVR_SA_PASSWORD_FILE=<(echo -n "$SA_PASSWORD") /opt/mssql/bin/sqlservr --setup $sqlservr_setup_options 2>&1 > /var/opt/mssql/log/setup-$$Then I tried to run /opt/mssql/bin/sqlservr manually and was able to reproduce the issue.
twright-msft
commented
7 years ago OK, now we seem to be getting somewhere... Can you please try a couple of things: 1) Try using a relatively simple complex password like 'Seattle100'. 2) If that doesnt work then try running with -it ... bash, but instead of statingn sqlservr directly try executing the sqlservr.sh script. It does some things in a container context that need to be in place before sqlservr can start correctly. If sqlservr.sh doesnt run end to end then sqlservr will not start properly.
BTW we are in the process of making a couple of improvements to handle situations like this better: 1) Removing the sqlservr.sh script in the Docker container scenario. The Dockerfile CMD will start sqlservr directly and sqlservr will do everything it needs to do on startup. 2) Outputting the logging to the stdout so it is easier to see.
thiagoalves
commented
7 years ago @twright-msft Thanks for the help
The simpler password didn't help much, as you can see below:
root@docker_linux_4:~# docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=Seattle100' --name matrix-debug -d microsoft/mssql-server-linux; docker logs matrix-debug -f
544b2eb10611111937988003ac72246bb2eece605a7bf61d26f08346afb61d4d
Configuring Microsoft(R) SQL Server(R)...
No journal files were found.
No journal files were found.
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.Regarding sqlservr.sh, this is exactly what I tried to do, as mentioned in my previous comment. The script fails on line 90. See the output below:
root@a2056f94df03:/opt/mssql/bin# ./sqlservr.sh
Configuring Microsoft(R) SQL Server(R)...
sqlservr: Debugger.cpp:578: static void Debugger::Print(const void *, unsigned int): Assertion `"A serious error condition has been encountered." == nullptr' failed.
find: '/proc/231/task/245/fd/28': No such file or directory
No journal files were found.
No journal files were found.
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
nohup: redirecting stderr to stdout
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.@twright-msft Any news?
BTW we are in the process of making a couple of improvements to handle situations like this better
Do you have any time estimates for this one?
twright-msft
commented
7 years ago CTP 2.1, mid-May is the target now.
jkasmann
commented
7 years ago I am seeing the same symptoms. 17.03.1-ce on CentOS 7.3. In my case it fails doing 'docker build'. Build from the same Dockerfile succeeds on boot2docker. Anyone find any workarounds?
Configuring Microsoft(R) SQL Server(R)...
No journal files were found.
No journal files were found.
ls: cannot access '/var/opt/mssql/log/errorlog*': No such file or directory
ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.txt': No such file or directory
ls: cannot access '/var/opt/mssql/log/SQLDu*.mdmp': No such file or directory
ls: cannot access '/var/opt/mssql/log/system_health*': No such file or directory
Microsoft(R) SQL Server(R) setup failed with error code 1. Please check the setup log in /var/opt/mssql/log for more information.This is an evaluation version. There are [166] days left in the evaluation period.
Unexpected signal encountered in host environment. Signal: 5 (Trace/breakpoint trap) at 0x7f0fa0419022
Dump control values set to default. Behavior controlled with /var/opt/mssql/mssql.conf
===============================================
Mini dump type: miniplus (2)
Capturing mini and full dump: true
===============================================
Capturing core dump and information for sqlservr (pid 17)...
Capturing dump with paldumper
Attempt to capture dump with paldumper failed. Reference core.sqlservr.17.temp/log/debug.log for details
Capturing dump with gdb
SQL Log Directory /var/opt/mssql/log
Packaging core dump and information into compressed files.
Core dump and information will be written to /var/opt/mssql/log/core.sqlservr.05_03_2017_19_25_23.17 and logs in /var/opt/mssql/log/core.sqlservr.05_03_2017_19_25_23.17.log
tynor88
commented
7 years ago I'm with exactly the same issues.
Host: unRAID 6.3.3 Docker: Docker version 1.12.6, build 6b644ec
Log: Configuring Microsoft(R) SQL Server(R)... sqlservr: vm.cpp:38: BOOLEAN IsPossiblyValidPalAddress(const void *): Assertion 'result' failed.
cat: /proc/22/sched: No such file or directory No journal files were found. No journal files were found. ls: cannot access '/var/opt/mssql/log/errorlog': No such file or directory ls: cannot access '/var/opt/mssql/log/exception.log': No such file or directory ls: cannot access '/var/opt/mssql/log/SQLDu.txt': No such file or directory ls: cannot access '/var/opt/mssql/log/SQLDu.mdmp': No such file or directory ls: cannot access '/var/opt/mssql/log/system_health': No such file or directory
tynor88
commented
7 years ago Any updates on this topic? :(
Gorbush
commented
7 years ago As per my understanding - it should be done in a different way - not mounting the host folder as /var/opt/mssql - but defining it as volume
tynor88
commented
7 years ago @Gorbush Can you provide an example or point me to some documentation?
Gorbush
commented
7 years ago tynor88
commented
7 years ago Still not working in CTP 2.1
tynor88
commented
7 years ago Still not working in rc2
We are getting the following errors on two ubuntu servers, one with
1.12.3-cs4and the other with17.03-ee:Output of docker info:
Output of docker version: