IRIS images do not start in DD on WSL2 backend

[eduard93]

I'm unable to start the demo with DD on WSL2 backend. Here's the error info. How can I fix that?

Versions:

PS C:\temp\irisdemo-demo-fraudprevention> docker version
Client: Docker Engine - Community
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:43:18 2020
 OS/Arch:           windows/amd64

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
docker-compose version 1.24.1, build 4667896
docker-py version: 3.7.3
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.0.2p  14 Aug 2018
PS C:\Users\eduard> [System.Environment]::OSVersion.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
10     0      19041  0
PS C:\temp\irisdemo-demo-fraudprevention> wsl -l -v
  NAME                   STATE           VERSION
* docker-desktop-data    Running         2
  Ubuntu-20.04           Running         2
  docker-desktop         Running         2

I'm integrating WSL with Ubuntu-20.04 image

Ubuntu has free space (other images do start):

eduard@ed-pc:~$ docker images
REPOSITORY                                     TAG                           IMAGE ID            CREATED             SIZE
deeplearning_image_similarity_py               latest                        1fb26abf97c0        2 days ago          1.5GB
intersystemscommunity/irispy                   healthaa                      2fe3941db50a        12 days ago         7.4GB
distr_iris                                     latest                        b9bde30e50ba        2 weeks ago         1.34GB
docker.iscinternal.com/intersystems/iris       2020.4.0L-latest              ac0aeaf8eef0        2 weeks ago         1.34GB
tensorflow/tensorflow                          latest-jupyter                2d87e2e84687        4 weeks ago         1.69GB
tensorflow/tensorflow                          latest                        539d0e818045        4 weeks ago         1.54GB
store/intersystems/irishealth-aa-community     2020.3.0AA.331.0              94cadcfe1c17        3 months ago        4.37GB
tensorflow/tensorflow                          1.15.2-py3-jupyter            3973570918d7        7 months ago        1.33GB
intersystemsdc/irisdemo-demo-fraudprevention   pos-version-1.7.0             bbd5998aab45        7 months ago        1.26GB
intersystemsdc/irisdemo-demo-fraudprevention   bankingtrnsrv-version-1.7.0   a8331dcbb90e        7 months ago        3.5GB
intersystemsdc/irisdemo-demo-fraudprevention   datalake-version-1.7.0        053fb573ab2d        7 months ago        3.09GB
intersystemsdc/irisdemo-demo-fraudprevention   bankingcore-version-1.7.0     002b5e616d0b        7 months ago        3.08GB
intersystemsdc/irisdemo-base-spark-iris        version-1.1                   65fcb5a68a0a        10 months ago       1.71GB
intersystemsdc/irisdemo-base-zeppelin          version-1.3                   d506554d232c        10 months ago       6.25GB
eduard@ed-pc:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd        251G  1.1G  238G   1% /
tmpfs           9.4G     0  9.4G   0% /mnt/wsl
/dev/sdc        251G   31G  208G  13% /mnt/wsl/docker-desktop-data/isocache
none            9.4G   12K  9.4G   1% /mnt/wsl/docker-desktop/shared-sockets/host-services
/dev/sdb        251G  117M  239G   1% /mnt/wsl/docker-desktop/docker-desktop-proxy
/dev/loop0      231M  231M     0 100% /mnt/wsl/docker-desktop/cli-tools
tools           465G  433G   33G  94% /init
none            9.4G     0  9.4G   0% /dev
none            9.4G  8.0K  9.4G   1% /run
none            9.4G     0  9.4G   0% /run/lock
none            9.4G     0  9.4G   0% /run/shm
none            9.4G     0  9.4G   0% /run/user
tmpfs           9.4G     0  9.4G   0% /sys/fs/cgroup
A:\            1020M   80K 1020M   1% /mnt/a
C:\             465G  433G   33G  94% /mnt/c
D:\             804G  802G  1.9G 100% /mnt/d
E:\             2.8T  2.6T  167G  95% /mnt/e
F:\             128G  126G  2.7G  98% /mnt/f
eduard@ed-pc:~$

Got these errors on IRIS containers:

bankingcore_1        | [INFO] Starting InterSystems IRIS instance IRIS...
bankingcore_1        | [INFO] Starting InterSystems IRIS instance IRIS...
bankingcore_1        | [INFO]
bankingcore_1        | Now attempting WIJ compare, ./IRIS.WIJ
bankingcore_1        |
bankingcore_1        | InterSystems IRIS Write Image File has blocks to be written, but it has an illegal format,
bankingcore_1        | We failed to fully restore the WIJ. The recommended action is to
bankingcore_1        | abort irisstart, resolve the problem and try again. The other
bankingcore_1        | option is you can delete the image journal and continue with irisstart.
bankingcore_1        | If you delete the image journal and continue you will probably have
bankingcore_1        | database degradation in the databases that were not fully restored.
bankingcore_1        |
bankingcore_1        | If you are not absolutely sure you want to do this, do not continue
bankingcore_1        | if you change your mind, you can run irisstart again and you will
bankingcore_1        | be given this choice again.
bankingcore_1        | Is an operator present to answer questions (Y/N)? Y =>
bankingcore_1        | Do you want to delete the Write Image File (Y/N)? N =>
bankingcore_1        | A problem was encountered while attempting to restore the write image
bankingcore_1        | journal file (IRIS.WIJ). See the console log for more information
bankingcore_1        | before attempting to start the instance again.
bankingcore_1        |
bankingcore_1        | ** Startup aborted **
bankingcore_1        |
bankingcore_1        | Starting IRIS
bankingcore_1        | Recovery failure. Startup aborted.
bankingcore_1        |
bankingcore_1        | [ERROR] Command "iris start IRIS quietly" exited with status 256
bankingcore_1        |
bankingcore_1        |
bankingcore_1        |
bankingcore_1        | *** Recovery started at Thu Aug 27 07:19:36 2020
bankingcore_1        |      Current default directory: /usr/irissys/mgr
bankingcore_1        |      Log file directory: ./
bankingcore_1        |      WIJ file spec: ./IRIS.WIJ
bankingcore_1        | Recovering local (./IRIS.WIJ) image journal file...
bankingcore_1        | Starting WIJ recovery for './IRIS.WIJ'.
bankingcore_1        |   0 blocks pending in this WIJ.
bankingcore_1        | WIJ pass # is 0.
bankingcore_1        |   WIJ file does not match expectations: imblkper was 127, not 127; or dirblks was 0 outside 1..127.
bankingcore_1        | Operator: "Is an operator present to answer questions (Y/N)? Y => ", answer defaulted: "yes".
bankingcore_1        | Operator: "Do you want to delete the Write Image File (Y/N)? N => ", answer defaulted: "no".
bankingcore_1        | [ERROR] Possible causes:
bankingcore_1        | [ERROR]  - InterSystems IRIS was not installed successfully
bankingcore_1        | [ERROR]  - Invalid InterSystems IRIS instance name
bankingcore_1        | [ERROR]  - Insufficient privilege to start InterSystems IRIS (proc not in InterSystems IRIS group?)
bankingcore_1        | [FATAL] Error starting InterSystems IRIS
bankingcore_1        | [INFO] Starting InterSystems IRIS instance IRIS...
bankingcore_1        | [INFO]
bankingcore_1        | Now attempting WIJ compare, ./IRIS.WIJ
bankingcore_1        |
bankingcore_1        | InterSystems IRIS Write Image File has blocks to be written, but it has an illegal format,
bankingcore_1        | We failed to fully restore the WIJ. The recommended action is to
bankingcore_1        | abort irisstart, resolve the problem and try again. The other
bankingcore_1        | option is you can delete the image journal and continue with irisstart.
bankingcore_1        | If you delete the image journal and continue you will probably have
bankingcore_1        | database degradation in the databases that were not fully restored.
bankingcore_1        |
bankingcore_1        | If you are not absolutely sure you want to do this, do not continue
bankingcore_1        | if you change your mind, you can run irisstart again and you will
bankingcore_1        | be given this choice again.
bankingcore_1        | Is an operator present to answer questions (Y/N)? Y =>
bankingcore_1        | Do you want to delete the Write Image File (Y/N)? N =>
bankingcore_1        | A problem was encountered while attempting to restore the write image
bankingcore_1        | journal file (IRIS.WIJ). See the console log for more information
bankingcore_1        | before attempting to start the instance again.
bankingcore_1        |
bankingcore_1        | ** Startup aborted **
bankingcore_1        |
bankingcore_1        | Starting IRIS
bankingcore_1        | Recovery failure. Startup aborted.
bankingcore_1        |
bankingcore_1        | [ERROR] Command "iris start IRIS quietly" exited with status 256
bankingcore_1        |
bankingcore_1        |
bankingcore_1        |
bankingcore_1        | *** Recovery started at Thu Aug 27 07:20:41 2020
bankingcore_1        |      Current default directory: /usr/irissys/mgr
bankingcore_1        |      Log file directory: ./
bankingcore_1        |      WIJ file spec: ./IRIS.WIJ
bankingcore_1        | Recovering local (./IRIS.WIJ) image journal file...
bankingcore_1        | Starting WIJ recovery for './IRIS.WIJ'.
bankingcore_1        |   0 blocks pending in this WIJ.
bankingcore_1        | WIJ pass # is 0.
bankingcore_1        |   WIJ file does not match expectations: imblkper was 127, not 127; or dirblks was 0 outside 1..127.
bankingcore_1        | Operator: "Is an operator present to answer questions (Y/N)? Y => ", answer defaulted: "yes".
bankingcore_1        | Operator: "Do you want to delete the Write Image File (Y/N)? N => ", answer defaulted: "no".
bankingcore_1        | [ERROR] Possible causes:
bankingcore_1        | [ERROR]  - InterSystems IRIS was not installed successfully
bankingcore_1        | [ERROR]  - Invalid InterSystems IRIS instance name
bankingcore_1        | [ERROR]  - Insufficient privilege to start InterSystems IRIS (proc not in InterSystems IRIS group?)
bankingcore_1        | [FATAL] Error starting InterSystems IRIS

[eduard93]

I would like to clarify that standalone IRIS images start successfully.

Also WSL does not impose CPU/Memory limits the way old HV backend did.

[amirsamary]

Hi! I just upgraded my windows box to use WSL 2. I am using the same version of docker as you. The only difference is that I picked Ubuntu 18.04 instead of 20.

Mine is working fine.

Maybe you are having this problem because you are using a version of Ubuntu that is not supported by InterSystems IRIS 2019.3 (the version of IRIS used by this demo)?

We are going to upgrade to IRIS 2020. We are just waiting for a more stable build.

Kind regards, AS

[eduard93]

How's your RAM consumption?

Moved to U18, things got better (no crashes) but now vmmem consumes all free memory ~18Gb and hangs the host.

Testing this workaround.

[amirsamary]

I will take a look. Mine was running ok. IRIS is limited to take just a portion of memory for global buffers. It is not automatic memory allocation. The Spark cluster was also limited, I guess to 1G per worker plus 1Gb for the master. The UI + the two IRIS boxes and the Zeppelin node should be taking just a little more than 1Gb of RAM. Do you know who is taking. The demo should not take more than 4Gb of RAM total.

Can you look what process is consuming most of the memory?

I will try again on my PC later today...

[eduard93]

Workaround works. I'm running only this demo with RAM limit at 8 Gb and RAM consumption grows to ~8Gb and stops after that.

Without 8 Gb limit the consumption grows further.