[Docker Desktop for Mac] - Support for running x86-64 binaries with Rosetta 2

[christophermclellan]

Tell us about your request Support Rosetta 2 for running x86-64 Linux binaries on Apple Silicon

Which service(s) is this request for? Docker Desktop for Mac

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Ensure that users can run their Intel-based workloads on Apple Silicon, using Rosetta 2

Are you currently working around the issue? Docker Desktop users can currently use QEMU for this

[pbednarcik]

Was anybody able to successfully run Oracle docker image provided by Gvenzl running latest docker (4.21.1 with latest rosseta)? Still getting: assertion failed [result.value != EEXIST]: VmTracker attempted to allocate existing mapping

@pbednarcik On Ventura, no, it fails like you describe.

On Sonoma beta 2, it appears to start up but then ends with "ORA-03113: end of file on communication channel". I assume that's not normal? I've never used these images before.

@atomicmattie - nope that's not normal. Seems like improvement in Sonoma 2. Hopefully one day it will work as for now I have to use Oracle via Windows installation in Parallels.

[dzuelke]

PHP with the OPcache extension still crashes, unfortunately:

rosetta error: futex(FUTEX_LOCK_PI_PRIVATE) failure: 35

[jjimenez]

Oracle now has a new container image for oracle 19 that is arm native.

see oracle container registry

I was able to run

docker run -d --name local_db container-registry.oracle.com/database/enterprise:19.19.0.0

and it took a while, but it setup the database and ran. No more need for colima on apple silicon for this.

🎉

If you run into a problem authenticating you have to log in with

docker login container-registry.oracle.com

which, yes, means you need an oracle.com login

[jjimenez]

Was anybody able to successfully run Oracle docker image provided by Gvenzl running latest docker (4.21.1 with latest rosseta)? Still getting: assertion failed [result.value != EEXIST]: VmTracker attempted to allocate existing mapping

see my message above about the new native image for oracle 19

[atomicmattie]

Hi @bryansh! Welcome to the bug from hell 😉

Update: I'm on Sonoma beta 6 (public beta 5, build 23A5328b) now and an earlier crash I mentioned has been resolved. 🥳

2. docker run --platform=linux/amd64 --rm -it alpine:3.5, apk update && apk add gcc, /usr/libexec/gcc/x86_64-alpine-linux-musl/6.2.1/cc1 crashes with "Segmentation fault"

This unblocks at least one older stack I had been unable to use with Rosetta.

Also,

@atomicmattie regarding #384 (comment)

I also realized that there's a third test in this thread that works on both Ventura and Sonoma with the requisite updates: docker run --platform=linux/amd64 hello-world. That was mentioned by @kingtistel above.

I've retested and it still doesn't work. Do you have ECI enabled?

@kingtistel Apologies! I missed that detail. I don't have access to ECI—just the team tier.

[ghost]

Just wanted to say that please make this feature available in prod and not keep it in beta. This has saved me tonnes of headache running amd64 containers on M2 Mac. I say this because the app says that Beta features can be discontinued without notice and need this feature. Thank you.

[joelteply]

I have gone through a variety of builds and had determined that cutting the proc down to 4 worked better than 1, as others had noticed, however this is not ideal so I kept investigating.

On my mac m1 (arch: amd64 through rosetta) where I'm doing a build of OpenSSL (gcc, make) and rust compilation, it seems that the whole culprit is VirtioFS. Ater switching to "gRPC FUSE" I have had zero issues where it hangs and pollutes the docker logs with file access problems. "osxfs (Legacy)" setting also caused a long hang.

I would therefore highly suggest not using VirtioFS until it's fixed, and if using osxfs, make sure it works for you because it didn't for me. gRPC FUSE setting is twice as fast for me and doesn't hang no matter the number of procs allocated. I was manually setting makeflags for proc units in both rust and make, in addition to specifying cpu count for docker run. I am no longer needing to do either.

Apple M1 Pro Docker Desktop 4.23.0 (120376) Engine: 24.0.6 Compose: v2.21.0-desktop.1

[mcabreb]

The 4.24.0 release of Docker Desktop solved my issues in M1 Silicon. https://docs.docker.com/desktop/release-notes/#4240

[apoca]

@mcabreb but you used platform: linux/amd64 ?

[Meligy]

I don't know if this helps anyone, but I recently had issues with WordPress and MySQL in docker compose - although the recent docker already solved my issues with MS SQL, and the issue for the WordPress turned out to be a bit different.

The key issue is that I had docker-compose installed via homebrew. And this did not work well. I had to uninstall that, and restart my terminal window, etc. for the docker-compose from docker desktop to be used, and work.

I also needed to upgrade the specific images. I know this suggests that newer versions have ARM support already and hence is not exactly this issue. I did this before the docker-compose, so, I assume doing the docker-compose thing first would have worked also.

[kingtistel]

Hi, I've come across a different issue: (Docker business support sent me here) If I have "Enhanced Container Isolation" enabled along side "Use Rosetta for x86/amd64 emulation on Apple Silicon" then the container does not start, emitting an error: rosetta error: Failed to get stack rlimit

So if I want the benefit of Rosetta I need to disable ECI.

Two steps reproduction

0. Make sure you have both ECI and "Use Rosetta for x86/amd64 emulation on Apple Silicon" enabled.
1. docker pull --platform=linux/amd64 hello-world
2. docker run --rm hello-world

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested rosetta error: Failed to get stack rlimit

Diagnostic ID: 59ACAA94-FE89-4B40-80A2-608E2C4632A9/20230620115220

After updating to macOS Sonoma and Docker Desktop v4.24 - this issue is resolved.

[mcabreb]

@mcabreb but you used platform: linux/amd64 ?

@apoca I used platform: linux/arm64/v8 to build my docker. My problem was by building a system that called node js code before compiling C++ in an M1. Without Rosetta it built the project but very slowly. With Rosetta it would always hung during the node build. So I had to do first a partial build without Rosetta, stop the container, activate Rosestta, build the rest of the system. With release v4.24.0 all the system builds with Rosetta, including the node js part.

[apoca]

I got this after upgrade to last mac OS and Docker version:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fffff384a93, pid=24880, tid=140737436714688
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x984a93]  oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x143
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/www/html/glamourosa/hs_err_pid24880.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp```

[wrohv]

@apoca I also see that error with macOS Sonoma and Docker 4.24.0. It wasn't failing in prior OS/Docker versions.

[marcdueker]

I got this after upgrade to last mac OS and Docker version:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fffff384a93, pid=24880, tid=140737436714688
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x984a93]  oopDesc* PSPromotionManager::copy_to_survivor_space<false>(oopDesc*)+0x143
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /var/www/html/glamourosa/hs_err_pid24880.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp```

Exactly the same issue here after upgrading to sonoma: M1 Max, Docker 4.24.0 and JDK 8-15 containers.

[MikhailPertsev1]

@marcdueker @apoca Did you guys figure out whats wrong and how to fix it?

[marcdueker]

@MikhailPertsev1 unfortunately not, I did a rollback to Ventura 13.6 which solves the issue. But there is also a separate thread for this issue: https://github.com/docker/for-mac/issues/7006

[gonzalopardogms]

@marcdueker is there a way to rollback to Ventura 13.6 other than "factory reset" of the macbook?

[marcdueker]

@marcdueker is there a way to rollback to Ventura 13.6 other than "factory reset" of the macbook?

As far as I know - no. I had to wipe everything to install Ventura.

[GarryOne]

@apoca , @marcdueker , and all others that are experience this problem: there is another open issue https://github.com/docker/for-mac/issues/7006

In the comments, you can see that there is a solution which works for many of us which suppose to set your Docker -> Resources -> CPU limit to 1 or 2 or even 4 worked for me.

[dinvlad]

Alas, I've been using this with 12 cores + Rosetta 2 for a long time, and have started experiencing this recently even on Ventura, without any changes or upgrades. Mine mainly manifests when compiling Go code (for example, I get error message runtime/internal/atomic: /usr/local/go/pkg/tool/linux_amd64/compile: signal: segmentation fault)

[GarryOne]

Latest release 4.25.0

https://docs.docker.com/desktop/release-notes/#4250

Rosetta is now Generally Available for all users on macOS 13 or later. It provides faster emulation of Intel-based images on Apple Silicon. To use Rosetta, see Settings. Rosetta is enabled by default on macOS 14.1 and later.

[MihaelaStoica]

Thank you all for your patience and support here, for testing new releases and letting us know what worked and what hadn't. As you have noticed, Rosetta is no longer a Beta feature in Docker Desktop.

There have been quite a few issues reported here, and most of them also confirmed here as fixed in Sonoma. The most recent issue, the SIGSEGV/SIGBUS error when running Java images, https://github.com/docker/for-mac/issues/7006, has also been fixed in the latest Sonoma version, 14.1.

We will close this roadmap issue soon, but if you encounter any more problems with Rosetta, please let us know by opening an issue in docker/for-mac

[jjimenez]

Thanks for the continued work across all of these issues @MihaelaStoica. I'll add a quick note that for Oracle, there is now an arm version of the image to work with, but running an amd image still doesn't work as expected.

docker run -d --name oracle_db --pull always --platform linux/amd64 container-registry.oracle.com/database/express:21.3.0-xe

still results in a container that doesn't run Oracle.

I'm still on Ventura, so that might be fixed on Sonoma.

I don't personally need this fixed right away, but it would be great as there is a licensing difference between the arm image that is available and the amd image that is available.

Congratulations on so much progress and so many improvements over time.

[MaLub]

@jjimenez do you have the Command for the ARM-Image? If I'm trying to pull the image, I'll only find the AMD-Version not the ARM-Version.

docker run -d --name oracle_db --pull always --platform linux/arm64 container-registry.oracle.com/database/express:21.3.0-xe
21.3.0-xe: Pulling from database/express
Digest: sha256:dcf137aab02d5644aaf9299aae736e4429f9bfdf860676ff398a1458ab8d23f2
Status: Image is up to date for container-registry.oracle.com/database/express:21.3.0-xe
docker: image with reference container-registry.oracle.com/database/express:21.3.0-xe was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.

[gonzalopardogms]

@MaLub I believe docker is not pulling the arm64 version because you already have an image matching the version (but not the platform). Try the following:

1. Go to docker desktop and search for express image under images. There you should identify the image and hopefully verify that it has a tag "AMD64" under it.
2. Remove/delete the image from docker desktop.
3. Execute again your command.

[jjimenez]

@jjimenez do you have the Command for the ARM-Image? If I'm trying to pull the image, I'll only find the AMD-Version not the ARM-Version.

docker run -d --name oracle_db --pull always --platform linux/arm64 container-registry.oracle.com/database/express:21.3.0-xe
21.3.0-xe: Pulling from database/express
Digest: sha256:dcf137aab02d5644aaf9299aae736e4429f9bfdf860676ff398a1458ab8d23f2
Status: Image is up to date for container-registry.oracle.com/database/express:21.3.0-xe
docker: image with reference container-registry.oracle.com/database/express:21.3.0-xe was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.

@MaLub, sorry if I'm not clear. The issue I'm exposing is that the amd image should be able to run under rosetta emulation and doesn't. There is an arm image for enterprise on version 19.19 (docker pull container-registry.oracle.com/database/enterprise:19.19.0.0)

You can use that if you follow the licensing requirements for enterprise

[cheema-corellian]

For the benefit of anyone who is wondering why the "Use Rosetta for x86/amd64" setting isn't showing up where everybody says it should be, it has been moved out of experimental section of the settings to the "General" section in the more recent versions of Docker Desktop. Make sure that it is turned on.

[marcosamorim]

I have a python container running from FROM --platform=linux/x86_64 python:3.10-slim-bullseye and it is getting stuck when I run the following: RUN npm install -g yarn && yarn install && yarn cache clean && yarn build:css && yarn build:js which only happens when I leave the Use Rosetta for x86/amd64 emulation on Apple Silicon enabled.

Anyone else noticing any weird behaviour after enabling that option? Already tried to build --no-cache, reinstall Docker app but none had success.

[agrawroh]

I'm on M1 (Sonoma 14.1) and Docker Desktop 4.25.1 (128006). I see this error while using --platform linux/amd64: _"rosetta error: bsssize overflow %"

[djcristi]

Thank you all for your patience and support here, for testing new releases and letting us know what worked and what hadn't. As you have noticed, Rosetta is no longer a Beta feature in Docker Desktop.

There have been quite a few issues reported here, and most of them also confirmed here as fixed in Sonoma. The most recent issue, the SIGSEGV/SIGBUS error when running Java images, docker/for-mac#7006, has also been fixed in the latest Sonoma version, 14.1.

We will close this roadmap issue soon, but if you encounter any more problems with Rosetta, please let us know by opening an issue in docker/for-mac

how about this one? https://github.com/docker/for-mac/issues/6998 . nextjs (npm build) with Rosetta on causes 1 cpu 100% and hang, and with rosetta unchecked it works but slow , macos 13.6.1 , tested with latest 3 docker for mac releases (4.24, 4.25.1 , 4.25.2)

[robert914]

Not sure if it's related, but this issue also highlights an issue with running 64-bit images on M1 Macs in Rosetta 2: https://github.com/docker/for-mac/issues/6796

Basically, if you run this command to show the CPU information:

docker run -it --rm --platform=linux/amd64 registry.access.redhat.com/ubi8/ubi:latest lscpu

If you run this on an Intel Mac, Windows or Linux machine, you'll see:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit

Which is what I would expect to see, both 32-bit and 64-bit op-mode support.

However, if you run this on an M1/M2 Mac with Rosetta enabled, you'll see:

Architecture:        x86_64
CPU op-mode(s):      32-bit

It's missing the 64-bit op-mode support. Why?

I'm wondering if all these various M1/M2 Rosetta 2 issues are stemming from the above missing 64-bit CPU op-mode support? I have some 64-bit images that will work fine on my M1 Mac, but then others will fail with strange segfaults or other kernel / disk access issues, but they all work fine on Intel machines.

[djcristi]

interesting , i've got this on, m1 , macos 13.6.1 , docker 4.25.2 Architecture: x86_64 CPU op-mode(s): 32-bit

[the-code-samurai-97]

I got the same error with M3 Max 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:05 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6031 arm64 and Docker version 24.0.7, build afdd53b I was trying to create a Docker with Ubuntu 22.04 LTS and install bazel

[spkane]

The switch to requiring users to use Rosetta instead of QEMU appears to have broken things pretty significantly, and there does not appear to be an obvious way to revert to the old QEMU-based behavior in Docker Desktop for Mac.

It is really easy to reproduce the core issue here with a test comment like this:

$ docker run --rm --platform linux/amd64 --name mongo bitnami/mongodb:6.0

Unable to find image 'bitnami/mongodb:6.0' locally
6.0: Pulling from bitnami/mongodb
e1386674fd12: Pull complete
Digest: sha256:fa089cf67d876b2be0f3903819f6579b45b090575ae48dc42a33fd0936412231
Status: Downloaded newer image for bitnami/mongodb:6.0

mongodb 20:36:18.66 INFO  ==>
mongodb 20:36:18.68 INFO  ==> Welcome to the Bitnami mongodb container
mongodb 20:36:18.70 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 20:36:18.72 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 20:36:18.73 INFO  ==>
mongodb 20:36:18.75 INFO  ==> ** Starting MongoDB setup **
mongodb 20:36:18.82 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 20:36:18.98 INFO  ==> Initializing MongoDB...
mongodb 20:36:19.18 INFO  ==> Deploying MongoDB from scratch...
/opt/bitnami/scripts/libos.sh: line 346:   196 Illegal instruction     "$@" > /dev/null 2>&1

Note the Illegal instruction error message at the end. You get this when you run almost any binary inside the container.

• Are there plans to allow people to revert to the old behavior until there is a fix that allows Rosetta to handle x86_64 properly?

Also, for the record, the lscpu output:

❯ docker run --rm --platform linux/amd64 --entrypoint lscpu --name mongo bitnami/mongodb:6.0
Architecture:                       x86_64
CPU op-mode(s):                     32-bit
Byte Order:                         Little Endian
CPU(s):                             8
On-line CPU(s) list:                0-7
Thread(s) per core:                 1
Core(s) per socket:                 8
Socket(s):                          1
Vendor ID:                          0x61
Model:                              0
Stepping:                           0x0
BogoMIPS:                           48.00
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; __user pointer sanitization
Vulnerability Spectre v2:           Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 asimddp sha512 asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp flagm2 front

[djs55]

@spkane thanks for the repro steps, we suspect Rosetta doesn't handle AVX instructions properly.

In the meantime could you try disabling Rosetta from the UI "Settings > General > Use Rosetta for x86/amd64 emulation on Apple Silicon". It should switch back to qemu after an "Apply & restart":

Note you may have to scroll down to see the option.

[dzuelke]

@spkane thanks for the repro steps, we suspect Rosetta doesn't handle AVX instructions properly.

No suspecting necessary ;)

From About the Rosetta Translation Environment | Apple Developer Documentation:

Rosetta translates all x86_64 instructions, but it doesn’t support the execution of some newer instruction sets and processor features, such as AVX, AVX2, and AVX512 vector instructions. If you include these newer instructions in your code, execute them only after verifying that they are available. For example, to determine if AVX512 vector instructions are available, use the sysctlbyname function to check the hw.optional.avx512f attribute.

[agrawroh]

@spkane thanks for the repro steps, we suspect Rosetta doesn't handle AVX instructions properly.

In the meantime could you try disabling Rosetta from the UI "Settings > General > Use Rosetta for x86/amd64 emulation on Apple Silicon". It should switch back to qemu after an "Apply & restart":

Note you may have to scroll down to see the option.

Yeah, that's what I ended up doing on my end to fix the error I saw here: https://github.com/docker/roadmap/issues/384#issuecomment-1814228995

[padrepitufo]

I was a bit surprised to find this thread (I really should have looked here first 🤦‍♂️ ) - regardless I had been dealing with it on this ticket https://github.com/GoogleCloudPlatform/cloud-sdk-docker/issues/356. The fix for me turned out to be either:

• what is mentioned here (disable Use Rosetta for x86..) or
• run softwareupdate --install-rosetta --agree-to-license and then re-enable Use Rosetaa for x86..

works on my machine 🤷‍♂️

[spkane]

The lack of AVX support in Rosseta 2 is something that is unlikely to change and the workarounds that Docker is providing appear to work around this well enough.

Either disabling Rosetta or using -e EXPERIMENTAL_DOCKER_DESKTOP_FORCE_QEMU=1 for containers that need AVX, etc.

[erharb]

I was a bit surprised to find this thread (I really should have looked here first 🤦‍♂️ ) - regardless I had been dealing with it on this ticket GoogleCloudPlatform/cloud-sdk-docker#356. The fix for me turned out to be either:

* what is mentioned here (disable `Use Rosetta for x86..`)
  or

* run `softwareupdate --install-rosetta --agree-to-license` and then re-enable `Use Rosetaa for x86..`

works on my machine 🤷‍♂️

Unfortunately the softwareupdate method is not working for me, disabling Rosetta is the only guaranteed solution at the moment for the specific containers that reproduce the problem consistently.

The slightly annoying thing is that it keeps re-enabling itself after each time a software update is taken.

[tomholford]

I had an issue with Rosetta + Docker pegging the CPU on Apple Silicon. For anyone encountering this issue with Vite, I found that installing @rollup/rollup-linux-arm64-gnu and setting platform: linux/arm64 in the docker-compose.yml helped resolve this issue.

[ps-hdavoodi]

@spkane thanks for the repro steps, we suspect Rosetta doesn't handle AVX instructions properly.

In the meantime could you try disabling Rosetta from the UI "Settings > General > Use Rosetta for x86/amd64 emulation on Apple Silicon". It should switch back to qemu after an "Apply & restart":

Note you may have to scroll down to see the option.

Thank you!

--platform linux/amd64 and this fixed my issue.

[mathisrome]

Hi guys,

I'm trying to make an image to be able to launch dedicated servers (Satisfactory in this case) via SteamCMD on a Mac M1 Pro.

For this I need a 64-bit Linux architecture.

However when I launch my image with the --platform linux/amd64 option I get the following error: ERROR! Failed to install app '1690800' (Requires 64bit operating system) When I run the lscpu command, the output returns: CPU op-mode(s): 32-bit.

Unfortunately when trying the solutions proposed above, none of them work.

Here are the steps to reproduce the bug: Launch a debian container with the command: docker run --platform linux/amd64 -it debian /bin/bash

And run the following command sequence:

export CPU_MHZ=2000 && apt-get update && apt-get upgrade -y && apt-get install software-properties-common curl wget unzip vim nano procps util-linux -y && apt-add-repository -y non-free && dpkg --add-architecture i386 && apt-get update && apt-get upgrade -y && apt-get install lib32gcc-s1 -y && mkdir steam-servers && cd steam-servers && curl -sqL "https://steamcdn-a.akamaihd.net/client/installer/steamcmd_linux.tar.gz" | tar zxvf - && ./steamcmd.sh +force_install_dir ./SatisfactoryDedicatedServer +login anonymous +app_update 1690800 -beta public validate +quit

Please remember to change the CPU_MHZ variable at the start of the command

[maxiedaniels]

I’m experiencing insanely slow performance using —platform Linux/amd64, but I have to because one of the Python libraries I’m using doesn’t support arm I guess. I’m confused about why it’s so slow though - is Docker not using Rosetta 2 on Mac silicon for some reason? I know Rosetta 2 emulates fairly quickly with everything I’ve thrown at it so I don’t know why things run at snails pace on docker with —platform..

[liuyehcf]

I can debug inside the Linux/amd64 container with the following steps:

# this will hang
ROSETTA_DEBUGSERVER_PORT=1234 ./main &

# gdb
gdb
(gdb) set architecture i386:x86-64
(gdb) file main
(gdb) target remote localhost:1234
(gdb) continue

But how can I analyze the coredump if the program crashed? I failed to find any doc related to this, does anyone konw how?