macOS 11 (Big Sur): Request to access Microphone

chjan commented 3 years ago

Tested from 11 of 20.11.20 to latest build DMG of 11.12.20. Same severe result:

Summary

Java apps on Big Sur are currently stuck IF system resources are used that are NOT defined as system default resource. One simple example is the built in mic of a MacBook Pro. If the mic is set to default in the system, Java can access it, else not.

The reason for this behaviour is that Big Sur requires the user to manually acknowledge "permission requests" sent by the apps. One example on an app that does this correctly is the Reaper DAW (WWW.REAPER.FM). One see the requests are lined up automatically as this app is installed. Look to Apple system preferences / security to observe this.

Java needs to do the same, or there must be an API that enables the app programmer to make such permission requests. This is really super high priority and a remedy should come in a short term release. At the moment, java apps on Big Sur are stuck (for the case that non-default resources of any kind is used).

Steps to reproduce

Just observe with mic as example that Java does not request permission, while Reaper (free to install) do request permission.

Expected results

Actual results

Triaging info

Java version:

What is your operating system and platform?

How did you install Java?

Did it work before?

Did you test with other Java versions?

aahlenst commented 3 years ago

Thanks for getting in touch. Your description is rather vague. It is unclear whether it is a problem of missing entitlements and it's also unclear whether some app has bundled Java or whether we're talking here about something that uses Java. If it's the latter, you need to raise this with OpenJDK, especially if new APIs need to be implemented.

Ideally, provide a small sample that reproduces the problem and describe the expected behviour.

aahlenst commented 3 years ago

chjan commented 3 years ago

Perhaps vague, but this is a problem FOR ANY JAVA APPLICATION that uses either openjdk or Oracles commercial versions, and the error is that the JDK/JRE does not request permission to use system resources that are not currently set as default in the operating system. One example is the microphone. If NOT set currently as default, then it cannot be accessed, because all these JDKs do not request permission to use the resource. And this is the point, that the later versions of OSX requires the user to MANUALLY grant such access. This takes the form that the operating system ask via notification "TERMINAL WANTS TO ACCESS MICROPHONE. ALLOW?" The user has to confirm and then things work. I used TERMINAL as an example because if you start a java app in terminal with (in our case) $Java -jar soundpimp.jar, then TERMINAL will embrace the java application and ask the above question (ONLY when the microphone is currently NOT set as default). Now compare JDK/JRE to TERMINAL (or REAPER, or...), with TERMINAL being Apple´s own line command tool: The developer behind TERMINAL and REAPER (a DAW) and almost all other programs has grasped this new rule. BUT THERE IS NO JDK/JRE that has grasped this rule!!!! It means java is partly stuck on Apple operating systems at the moment. So, OpenJdk, Oracles JDK , all the JREs, etc, etc. They need to ask the apple operating system for allowance. GOT IT? Something that is unclear? OK, please blame it on this hopeless editor :-) But send me an email if you need more info. chj@hdsoundlab.com. Merry Christmas.

aahlenst commented 3 years ago

Let me explain... I'm not looking for caps lock nor qualifications like major showstopper. I'm looking for a technical, actionable description of the problem: What exactly you're trying to do, the behaviour you're expecting, what happens instead, maybe garnished with some pictures and links to Apple's docs. Please have a look at https://github.com/AdoptOpenJDK/openjdk-support/issues/228 and https://github.com/AdoptOpenJDK/openjdk-support/issues/146. Both a great examples of actionable enhancement requests. The more homework you do, the bigger the chance that not only something happens but also the right kind of something.

chjan commented 3 years ago

That's ok. I am not either, but this "lack of a mandatory feature" seems to me like nothing else if not a major showstopper, and it is kind of hard to find a place to report the problem. Your site focusing on AdoptOpenJDK... is one option, but if you look to Oracle with their commercial variants, where is the bug report feature? I am using their Facebook message feature and got some attention therein, however, it is an unsatisfactory situation.

So, I can attach a picture if you like. The picture tells you that the following applications, used as examples, do the right thing, because they request permission for a system resource, a microphone in this case. You can observe that more apps are lining up in compliance, only java has not grasped the new game.

What is not so obvious, but still true, is that when you as a user execute a java application as a .jar executable, by double-clicking it, neither the app (my own) nor the JDK/JRE is posing the mandatory request for (in this example) permission to use the microphone of the laptop. And this is mandatory due to the brand new ands built-in security mechanisms of the Apple OS, as I said: The user of the machine must make the choice to allow system resources to be used or exploited by this or that app that the user him/herself probably has started. This is not to be confused with the traditional setup of a mic inside a java application.

So then I repeat: The applications TERMINAL and REAPER are compliant to these new security mechanisms, and you can observe that these two apps already has provoked the user to "CHECK" the said allowance for microphones. And as I discuss this here, I in parallel install new stuff like PARALLEL and some EQ app. You see also that this is the "security" part of the "system preferences" of the Big Sur version of the Apple OS (i.e. latest version).

So the question you and we could ask, supposedly as entities that would like Java to work flawlessly also on an Apple computer, is this: Why does the JDK/JRE not pose that question for mic allowance?

To our knowledge, there is no new "api" that would make it possible to initiate such permission request from the applications, so it must be done by those working with the R&D of the JDK/JRE, and sorry for capital letters: WHEREVER ANYONE WORKS WITH JAVA ON APPLE OPSYS, including but not limited to you and your team.

chjan commented 3 years ago

I must add one more info: When the allowance is not yet granted by the user, there is no error report, no error notification, no alarm, nothing. It just stops working, thus creating havoc. Mic does not work unless it is currently the default audio input device. This is for java. For Reaper, due to allowance granted, it works. It is quite normal for some applications that the mic is not currently default (due to something called audio routing, which its a bit advanced).

And even more: One could imagine that the operating system would have been so kind as to ask the user this vital question for allowance, but as you can observe in the screenshot above, no such option is available to manually choose additional apps. Therefore, the whole scenario is a showstopper at the moment.

aahlenst commented 3 years ago

You write that you're having problems with your own app, so do I assume correctly that you are a Java developer? If yes, can you write a main() program that accesses the microphone and does not work? Can you paste that here?

When you start your application in Terminal, do you get any output there from the system or JDK? Can you have a look into macOS's system console whether you can get any info there? Can you paste it here? Does it look like https://bugs.openjdk.java.net/browse/JDK-8244951?

Can you try with the macOS build from http://jdk.java.net/17/? Does it fail, too?

Do you have a link to the Apple documentation that describes exactly what kind of API needs to be implemented?

chjan commented 3 years ago

JDK17? Still not working, but thanks for the link with clues regarding how to file bugs. I was at oracle Java pages and no link to that JDK site there.

Yes, it kind of resembles that bug linked, but not bulls eye as I understand it. My report is talking about a general problem in how the whole Java community handles the new security mechanisms in Apple Op-sys.

I do not have link to Apple documentation. Although our Java product SoundPimp has been on the market for a decade, customers on Apple are not that many, so we are not experts on Apple.

I attach two more screenshots. One showing that there is a "developer mood" identified in the Apple security setup, however, it does not work as remedy for this problem. Secondly, you also see that SOME of the lines in the system resource selector has the + - option to add applications that shall be allowed. And then you see I have added Java JDK from the /bin folder. Not working. But you have also seen that some system resources, like mic, does not have that + - option. I have no idea why.

chjan commented 3 years ago

The second screenshot show a run of our java app SoundPimp inside Netbeans on this Apple Big Sur machine. Observe the log says everything ok. IOW, java does not discover that there is in reality no sound. You see upper right that the "mic" BLACKHOLE has been selected as system default. Then you see lower left that BLACKHOLE has been selected as input to our SoundPimp, that does its DSP algorithm and sends to output which has been selected as MacBook speakers. BUT, there is no signal running through soundpimp, because the peak meter just above the REW word, in the black section of the SoundPimp dashboard, is empty. IOW, the Op-sys just cut the stream without further notification, probably as a result of the fact that Java does not request permission. This is only my best guess regarding cause.

I must also correct myself. I talked earlier about SYSTEM DEFAULT RESOURCES, and that those are allowed. I think that was too hasty conclusion. It is more likely that e.g. speakers (=output) is just not regarded as a security threat, and therefore is allowed to be accessed, while mic most certainly is a security threat and thus enabled on the given conditions, which is manual allowance by the user, after programmatic request posed by the app that needs that, e.g. Java (soon :-))

Finally, what you also see on this screenshot, is yet another java app from us, namely a radio streamer. It is being used as a source of audio stream to test this setup. And importantly, when you upper right select MACBOOK SPEAKERS as SYSTEM DEFAULT, then this radio app works. Sound is heard. The reason is of course that the whole chain of Java based audio is short-circuited.

Per coincidence, we are most likely creating a small android app after Christmas that will access microphone. We can then create that little app with mic also for Java. We use more and more Android Studio as IDE also for pure Java projects, so it will be possible.

Also note that Netbeans does not work as "embracing wrapper" for soundpimp in a similar manner as I already explained that TERMINAL does. So Netbeans is subject to the same problem as running a java app as autonomic executable. Whether even Android Studio is on this list, we will find out when we make that java app with a mic input.

I strongly prefer email chj@hdsoundlab.com for further and more detailed discussions. We are bordering on confidentiality regarding our products on this thread. Please, for your consideration, and then perhaps with a final conclusion announced here in this thread, when available.

To me, it looks like all those responsible for JDK things in the java community need to discuss this problem in general and dig out a solution. Again, my guess only at this stage.

tresf commented 3 years ago

It is unclear whether it is a problem of missing entitlements

The entitlement appears to be called NSMicrophoneUsageDescription. This was previously an iOS-only feature which is why it's hard to find the exact documentation link for macOS (at the time of writing this, most search results land on the wrong docs). OpenJDK seems to already have this entitlement in JDK16 and JDK11, so I don't think there's any changes needed. Please correct me if this is wrong.

if new APIs need to be implemented.

It appears to be called AVCaptureDevice authorizationStatus(for:) [api mirror]. Since this is new with 10.14, openjdk will have to be enhanced with this functionality (assuming .plist changes won't make a difference ).

switch AVCaptureDevice.authorizationStatus(for: .video) {
    case .authorized: // The user has previously granted access to the camera.
        self.setupCaptureSession()

    case .notDetermined: // The user has not yet been asked for camera access.
        AVCaptureDevice.requestAccess(for: .video) { granted in  // change to audio, I assume?
            if granted {
                self.setupCaptureSession()
            }
        }

    case .denied: // The user has previously denied access.
        return

    case .restricted: // The user can't grant access due to restrictions.
        return
}

Unlike other security features in macOS, it appears there's no manual workaround. Testing on Big Sur, the Microphone area has no option to manually add an application, so I would make an educated guess that the missing API is a hard-stop.

Note, testing on an M1, I can't get access to the Microphone at all, which contradicts the OP's "Default device" observations. This may be a difference in default behavior between Intel and M1 to ease the transition into a stricter security model, but that's speculative.

Because I can't find a way to obtain any input from the Microphone at all, I have no viable code to share. On the M1, the OS seems to simply block access to the device. I can only list details about playback devices, not Microphones or input devices.

tresf commented 3 years ago

@chjan can you please change the title to be more descriptive, e.g. "Request to access Microphone"? Oddly, I have code which reads the screen and the prompt raises just fine, I assume because the Security manager has an option to raise this request so blindly calling this "system resources" is too vague and should be more specific. Also, words like MAJOR should be removed (subjective) and instead the maintainers can set priority through their own triaging efforts (which is volunteer, mind you).

chjan commented 3 years ago

OK, sorry for wrong use of words, but I had already tried to report this here and there and could not even find a place at Oracle to report this problem. I allow your rephrasing according to your own convenience. I also respect your volunteer mindset for sure. But, we all want java to work, so lets make that happen. Our situation is that customers reports java stopped working after op-sys update, and that is serious business, however not related in particular to openjdk, but java in general. Our products are applications on the market for a decade. I will return tomorrow, busy this evening.

tresf commented 3 years ago

customers reports java stopped working

This is a volunteer-led forum. For immediate support purposes, please reach out to a commercial JDK support provider.

If you want to take your chances simply submitting the bug upstream, that can be done here: https://bugreport.java.com/bugreport/start_form.do. Note, this process at java.com is NOT a transactional system. Your bug report won't be immediately accepted, but it should email you in days (or weeks) IF it is accepted.

lets make that happen.

This isn't a pep-rally, it's a technical bug report. Please provide technical details that move this forward.

But, we all want java to work

There are many issues with Java on recent Macs and audio is only one of them, and possible on the lower-priority list for application developers currently. This isn't to minimize the problem and how it affects people, but rather help you and others understand that the list of bug plaguing Big Sur and openjdk are long winded. To get this bug fixed before others will require someone with the technical knowledge of the problem and incentive to look into it to submit a bug report and a subsequent patch to openjdk, which can be a lengthy process even after the code is authored and submitted. Then, backporting it to Java 8 or 11 can take a while too. Blindly stating enthusiastic phrases like "we all want java to work" doesn't help this process, but rather wastes time of the volunteers that read each bug report that arrives.

chjan commented 3 years ago

It will not be helpful if you continue to pick one of my phrases and make a statement to that. This looks to be not a particular problem for openjdk, but for java in general. I say that because no JDKs from any java runtime developers is at the moment working.

This is probably not a "mic" only problem. I anticipate that the exact same problem will arrive if you try to use built in camera and other system resources that in any way can be related to a security threat, as seen from the developers of the Apple opuses. For your consideration.

Thus, your comment 2 days ago may be on to something that will grant remedy, but it is probably incomplete since it is a general problem.

To that comment, I repeat that my analysis concerning "default device" probably was too hasty. Rather, the rejection is probably related to the fact that MIC is one of many examples on system resources that can be a security threat.

Thanks for the link to where bug reports can be filed. It was because I could not find that link on the java section of Oracle´s webpage, that I decided to file it here in an attempt to get some attention to what I (still) perceive as a showstopper. The correct place would have been that link, yes.

If the java compliance to Big Sur is longwinded, then my suggestion (not being part of the openjdk team) would be that these challenges need to be overcome. Further to that, since the problem is for all JDKs at the moment, I stated above that perhaps it would be a good idea for those working with JDK variants to "come together" and identify the list of malfunctions. This is just a suggestion. It could be that Oracle should do that, and then you can follow the route they declare. It must be done also for openjdk, else, why deliver openjdk? (rhetorically asking).

IMPORTANTLY, yesterday I installed audio drivers for RME Digiface USB, a sound card of top quality and that includes the accompanying software and drivers. Doing that, I as a user got a series of security related explanations and questions. Regarding java JDKs (and JREs), I ask if this is could be a very good example on the line of questions that the user would get also when installing any JDK? I will let the screenshot speak for themselves. Kind of self-evident stuff in this new context of security mechanisms on Big Sur. However, this is not a dialog that pops up on a specific system resource, it is the dialog related to the install of the drivers. However again, this could be a lead regarding how openJDK must also pose such questions to the user. Simply a suggestion for your consideration.

Those working with JDKs on Apple will then need some mandatory technical papers that outline some API that all software manufacturers must read, understand and comply to. And if RME can get hold of such information, why can't you? Why can't Oracle? Obviously, there is a need to study this API, more or less as a first step. Compliance would then probably be general and solve things in the context of java, not in the context of Mic only. Agree?

I also mentioned that there is presented in System Preferences / security an option to manually allow anyway certain Developer Tools that does not comply to these security mechanism to operate as normal. I did not get that to work, but this COULD be a quick remedy that would allow openJDK to work in the short timeframe, while you work on a more compliant solution. The question then is which java file that shall be selected and defined here? I am very interested in your response on that, but if you cannot find a way neither, then there is perhaps the option to file a bug to Apple on that feature and state that it does not work for java. I attach a screenshot for this "developer tools" including my manual selection of java that did not work.

chjan commented 3 years ago

And just to rule out any false assumptions that could become time-thieves in this process, I installed the Parallels virtual Windows desktop, which is kind of impressing as it enables to run Windows .exe files in an Apple integrated manner. (Because I am moving from Windows to Apple for R&D and other stuff). On this virtual Windows, I have installed JDKs and it works more or less flawlessly. All the security problems in this thread are not present when in "windows" mode. Our applications works as normal including but not limited to "mic" access.

IOW, the problems is not related in any way to the hardware and the machine. I am on a 15.6 MacBook Pro Big Sur. We also have older Macbook pros from 2011, 2013 and 2016, and depending on op-sys version installed, those work as they should with Java.

aahlenst commented 3 years ago

All the security problems in this thread are not present when in "windows" mode. Our applications works as normal including but not limited to "mic" access.

Yes, because the applications are effectively running in Windows. Parallels acts as an arbiter between Windows and macOS and requests the necessary permissions for the VMs, not for the individual applications.

If the java compliance to Big Sur is longwinded, then my suggestion (not being part of the openjdk team) would be that these challenges need to be overcome.

Those suggestions aren't helpful because things aren't that simple. And it's neither the time nor the place to go into details. I'm going to hide posts that do not add technical value to this discussion to reduce noise. To clarify:

AdoptOpenJDK as a project is not in the business of developing OpenJDK, we're in the business of building binaries and providing associated tools so that's easier for end-users to use OpenJDK. The place to develop OpenJDK is OpenJDK.
If you are heavily impacted by a problem like this one here and feel that the wheels aren't turning fast enough, you have two options: (1) Purchase support from Oracle, IBM, BellSoft, Azul, ... This might get the problem the necessary attention you're looking for. (2) Volunteer to develop the necessary patches yourself. There are a lot of friendly people willing to provide assistance, but ultimately you have to do the work.

tresf commented 3 years ago

not a particular problem for openjdk, but for java in general.

Since Java 11, Oracle Java and OpenJDK are essentially the same product. The history is complex, it involves the deprecation of certain proprietary parts and creating open source alternatives where possible, but Oracle and the OpenJDK project have done just that.

For that reason, for most applications, "openjdk" and "java" are synonymous, quoting:

"Also, traditionally “commercial features” such as Flight Recorder, Java Mission Control, and Application Class-Data Sharing, as well as the Z Garbage Collector, are now available in OpenJDK. Therefore, Oracle JDK and OpenJDK builds are essentially identical from Java 11 onward."

So a problem with Java *IS* a problem with OpenJDK. I want to make sure that's as clear as possible.

This is probably not a "mic" only problem. I anticipate that the exact same problem will arrive if you try to use built in camera and other system resources that in any way can be related to a security threat

At time of writing this, the Camera isn't supported, which is why this hasn't been reported yet. Since projects like JavaCV are wrappers around other projects, getting them to comply with the enhanced security requirements will fall upon the 3rd party library maintainers (or their dependencies, such as OpenCV, ffmpeg). They won't be solved by Java directly. For this reason, it's best to research and provide evidence, not speculation. Thanks for changing the title to reflect the Microphone.

I as a user got a series of security related explanations and questions. Regarding java JDKs (and JREs), I ask if this is could be a very good example on the line of questions that the user would get also when installing any JDK?

Java used to be a "system wide" dependency for apps, but when the module system was introduced, Oracle started encouraging it to be distributed WITH applications putting the onus on a security dialog up to the application developer(s). What you observe is still a bug, but recommending the overreaching security dialogs for a system-wide framework is unlikely to happen as Java -- by design -- is moving away from this installation model.

[Big Sur] challenges need to be overcome

This is another blanket statement describing monumental amounts of work. In short, they are, but unless working with a commercial support provider, this enthusiasm is (again) misdirected. This is a volunteer effort to DISTRIBUTE Java. I'd be happy to share my experiences with the bug reporting process privately if you wish (e.g. over on the Slack channel).

these security mechanism to operate as normal. I did not get that to work, but this COULD be a quick remedy that would allow openJDK to work in the short timeframe

Me neither, as identified here, quoting:

Unlike other security features in macOS, it appears there's no manual workaround. Testing on Big Sur, the Microphone area has no option to manually add an application, so I would make an educated guess that the missing API is a hard-stop.

I strongly prefer email chj@hdsoundlab.com for further and more detailed discussions. We are bordering on confidentiality regarding our products on this thread. Please, for your consideration, and then perhaps with a final conclusion announced here in this thread, when available.

Please don't ask volunteers email bug reporters. GitHub can dispatch emails and you're already subscribed. That should be sufficient. The private information you are screenshotting and sharing wasn't requested but instead was submitted on your own behalf. You may delete anything personal (at any time) by editing those posts. If I'm wrong and if it WAS suggested in a message that you needed to provide personal information to resolve this bug, please point that out so we can improve our interactions moving forward.

Those working with JDKs on Apple will then need some mandatory technical papers that outline some API that all software manufacturers must read, understand and comply to. And if RME can get hold of such information, why can't you?

This post quotes the technical API. Integrating it into OpenJDK is a much larger effort, which is explained here.

As an aside, it's important -- when dealing with volunteer led projects -- to be objective and not accusatory or obligatory. Accusations assume warranty and reliability. Obligations assume support. There are many channels which offer this type of support and I'd be happy to talk in private about my personal experiences with these, but for the bug report, it's most valuable to stick to the technical details. Most of what's provided (these responses included) don't add useful information to this bug report and as a result may have the inverse effect and drive volunteers away from it. 🍻

aahlenst commented 3 years ago

So, AVCaptureDevice isn't known to OpenJDK: https://bugs.openjdk.java.net/issues/?jql=text%20~%20%22AVCaptureDevice%22. I also read through a ton of issues with the term "Microphone". https://bugs.openjdk.java.net/browse/JDK-8212558 is the closest but predates Catalina and Big Sur. So someone needs to report that upstream. Everybody can do that on https://bugreport.java.com/bugreport/, but please read https://docs.oracle.com/en/java/javase/15/troubleshoot/submit-bug-report.html beforehand. Sample code to reproduce the problem is mandatory. Otherwise it will be closed right away.

@chjan Maybe a crazy idea, but worth a try: Depending on your program flow, you could try calling into native macOS APIs with JNA to trigger the permission dialog.

kovdb commented 3 years ago

We had a similar issue with our java software.

We were not requesting access to a microphone, but to another application. This also needs to be granted first in the Automation section of the Privacy tab.

The access was not requested by macOS and it consequently did not work. We could fix this by packaging our application in a proper Application Bundle with a proper plist, application icon, correct folder structure...

Once we did this, macOS was requesting proper access on first use and our application was added automatically to the list in the Automation section (with the correct icon and application name).

tresf commented 3 years ago

packaging our application in a proper Application Bundle with a proper plist, application icon, correct folder structure...

This reminds me a bit of a bug I experienced with a mime launcher and a missing .plist entry. @kovdb Can you offer any insight as to which setting fixed this for your app so that the OP can propose this change to the affected project(s)?

chjan commented 3 years ago

We had a similar issue with our java software.

We were not requesting access to a microphone, but to another application. This also needs to be granted first in the Automation section of the Privacy tab.

The access was not requested by macOS and it consequently did not work. We could fix this by packaging our application in a proper Application Bundle with a proper plist, application icon, correct folder structure...

Once we did this, macOS was requesting proper access on first use and our application was added automatically to the list in the Automation section (with the correct icon and application name).

Thanks. That is valuable information. We normally do, but our last update using this method was a few years ago, and our latest attempt was with the single jar executable only. I will report it here if packaging will work for us. Date not defined for that process. Anyway, my question now is this: Will openJDK report this as a "bug" somewhere upstream, or are I or others expected to do that?

aahlenst commented 3 years ago

Will openJDK report this as a "bug" somewhere upstream, or are I or others expected to do that?

Given the (lack of) information it does not make sense if we do it. I have zero knowledge in packaging apps for the Mac and therefore cannot drive the necessary discussion.

If there's someone who has experience in packaging apps on the Mac, the best course of action is to mail https://mail.openjdk.java.net/mailman/listinfo/awt-dev with a reproducer, expectations and links to APIs. But be prepared that you have to come up with a patch (and sign the OCA beforehand).

@chjan I recommend to turn to a OpenJDK support provider: https://adoptopenjdk.net/support.html (IBM successfully delivered patches for macOS in the past), BellSoft might also be in a position to help. They'll also help with backports.

aahlenst commented 3 years ago

Closing due to lack of activity. We're also the wrong place to deal with this problem. If anyone wants to pursue it, please pick it up and post links to OpenJDK tickets etc.

tresf commented 3 years ago

It is unclear whether it is a problem of missing entitlements

The entitlement [appears to be called NSMicrophoneUsageDescription]

Just found it in AdoptOpenJDK's Info.plist at the very bottom. @chjan, does this help?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>CFBundleDevelopmentRegion</key>
    <string>English</string>
    <key>CFBundleExecutable</key>
    <string>libjli.dylib</string>
    <key>CFBundleGetInfoString</key>
    <string>AdoptOpenJDK 11.0.10+9</string>
    <key>CFBundleIdentifier</key>
    <string>net.adoptopenjdk.11.jdk</string>
    <key>CFBundleInfoDictionaryVersion</key>
    <string>7.0</string>
    <key>CFBundleName</key>
    <string>AdoptOpenJDK 11</string>
    <key>CFBundlePackageType</key>
    <string>BNDL</string>
    <key>CFBundleShortVersionString</key>
    <string>1.0</string>
    <key>CFBundleSignature</key>
    <string>????</string>
    <key>CFBundleVersion</key>
    <string>11.0.10</string>
    <key>JavaVM</key>
    <dict>
        <key>JVMCapabilities</key>
        <array>
            <string>CommandLine</string>
            <string>JNI</string>
            <string>BundledApp</string>
        </array>
        <key>JVMMinimumFrameworkVersion</key>
        <string>13.2.9</string>
        <key>JVMMinimumSystemVersion</key>
        <string>10.6.0</string>
        <key>JVMPlatformVersion</key>
        <string>11.0.10+9</string>
        <key>JVMVendor</key>
        <string>AdoptOpenJDK</string>
        <key>JVMVersion</key>
        <string>11.0.10</string>
    </dict>
    <key>NSMicrophoneUsageDescription</key>
    <string>The application is requesting access to the microphone.</string>
</dict>
</plist>

chjan commented 3 years ago

Yes, it is a question about entitlement and signing that will make a java app that needs audio input to run. And the entitlement is https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_device_audio-input

This thread is closed, but I would say that for anyone trying to make this work, there should be a teaspoon description somewhere, including but not limited to entitlement, signing, packager, etc. Such a description will make it MUCH easier for java developers to cope with new security rules on MacOs.

Who would be responsible for creating that, I have not succeed in finding. But anyone concerned about AdoptOpen JDK and Java survival on Mac systems should hear a sound in the alarm bell.

I have created an apple developer id and will take it from there. (soon). I am not sure yet which packager tool should be used. The built-in jPackager (I believe in jdk 15 and upwards?) seems to present a maturity in alpha stage. The usability of it is slim. Very slim. Should be worked on with high priority and be a part of the major description when presenting "how to use Java on Mac Os Big Sur and onwards". IMHO.

tresf commented 3 years ago

So I decided to dig into this a bit more... It turns out Java works quite fine with the Microphone access, but it's quite unintuitive (not to the fault of Java, by the way)

It asks once AND ONLY ONCE to access the Mic.
- Hitting escape is the same as clicking "No"
- There's no obvious way to re-trigger the request once its been answered (System Preferences, Security & Privacy, Privacy, Microphone has no option to remove the request)
- Java won't yell at you if you click no, it will just record silence

Just observe with mic as example that Java does not request permission, while Reaper (free to install) do request permission.

@chjan I find this statement to be incorrect. If you know of a way to re-raise this dialog, great, otherwise, this is an Apple bug. Below's my code. If there's something wrong with it, please be as specific as possible. Furthemore, if you're only experiencing this "bug" when bundling Java with your application, then this is more of a support request for help packaging and should probably be reworded to reflect that. (Something of which I'm also vested in sorting out).

Click to see code...

```java import javax.sound.sampled.*; import java.io.*; public class SoundRecorder { private TargetDataLine line; public SoundRecorder(File wavFile, long duration) throws LineUnavailableException, IOException{ new Thread(() -> { try { Thread.sleep(duration); } catch (InterruptedException ignore) {} line.stop(); line.close(); System.out.println("Finished"); }).start(); record(wavFile); } private void record(File wavFile) throws LineUnavailableException, IOException { AudioFormat format = new AudioFormat(16000, 16, 1, true, false); DataLine.Info info = new DataLine.Info(TargetDataLine.class, format); line = (TargetDataLine) AudioSystem.getLine(info); line.open(format); line.start(); // remove the old file if exists wavFile.delete(); System.out.println("Start capturing..."); AudioInputStream ais = new AudioInputStream(line); System.out.println("Start recording..."); AudioSystem.write(ais, AudioFileFormat.Type.WAVE, wavFile); } public static void main(String[] args) throws LineUnavailableException, IOException{ new SoundRecorder(new File(System.getProperty("user.home") + "/Desktop/test.wav"), 5000); } } ```

chjan commented 3 years ago

@tresf, I shall test your code. I assume you then created an executable jar, and that you run this by double-clicking it in its folderk. IOW you did not run it from terminal, and you did not run it from your IDE. Correct?

I am trying to make a point of the current situation on MacOs, as follows: It is not a Apple bug. It is Apple that has launched a new level of security in the later versions of the operating system. This requires a java jar executable to be packed as what Apple calls an app (a folder structure), which MUST include relevant entitlements AND signing of that app (so avoid hacker manipulation of it later on). Else it will not run! This makes perfectly sense. It is a good solution that brings more security to MacOs devices.

This is not java's fault. This statement is correct, but the "java community" needs to take this seriously and be eager to cope with this new situation. Therefore, my suggestion to make a description of how to pack a java jar executable as an Apple app, is a very good one. Because this is what many java developers MUST do when preparing their product for Mac Os. (Note: I have other java jar apps that runs as normal, simply because they don't contain usage that is related to security and entitlements).

Why? For one thing, Apple don't care one bit if Java applications work or not. They have even removed some of the tools needed to create the packaged app in Xcode IDE, as Java is down and out from their perspective.

THERE IS NOT OTHER WAY for the Java community but to cope with the situation and make it easy for java developers to launch their product on Mac Os. I am not in the slightest doubt that this must include a packager tool that is included in the JDK, just as is the case today with jpackager, a tool that unfortunately hold a much too low quality. To be honest, it is the worst java package I have ever encountered. And therefore, java developers are stuck, and no other entity on earth cares one bit.

This is the current game for Java on MacOs. There is no way around that. And if noone takes responsibility, then each developer must find his own solution, and that aint the winning team.

I am therefore worried that this thread almost completely lacks a response that could make me think that "someone" as grasped this situation and understands that SOMETHING HAS TO BE DONE TO RECITIFY IT.

I am sorry for capital letters, but I have now tried to explain this several times, in vain.

I return after testing your test app.

NOTE: Can you try to run this in Netbeans? You will then experience "silence" because Netbeans developer has not grasped this situation. Therefore, try to run it in AndroidStudio/Intellij, and it will work, because the team developing the latter has grasped the situation. TRY!

tresf commented 3 years ago

Requoting:

if you're only experiencing this "bug" when bundling Java with your application, then this is more of a support request for help packaging and should probably be reworded to reflect that. (Something of which I'm also vested in sorting out).

You replied:

This requires a java jar executable to be packed as what Apple calls an app (a folder structure), which MUST include relevant entitlements AND signing of that app

Sounds like your issue is specifically about packaging.

chjan commented 3 years ago

Requoting:

if you're only experiencing this "bug" when bundling Java with your application, then this is more of a support request for help packaging and should probably be reworded to reflect that. (Something of which I'm also vested in sorting out).

This goes very slow: I have not announced anything about bundling Java with my application as being a problem. This is simply part of packaging. I am talking solely about one app that uses security stamped system resources and that therefore needs an "apple app".

You replied:

This requires a java jar executable to be packed as what Apple calls an app (a folder structure), which MUST include relevant entitlements AND signing of that app

Sounds like your issue is specifically about packaging.

What I in vain have tried to communicate: It is not MY problem, it is everyones problem that wants to distribute their java app to the Mac Os platform. Please study my text much more thorough. Thanks!

tresf commented 3 years ago

I have not announced anything about bundling Java with my application

If you're using the system-wide Java, it actually simplifies this quite a bit; the entitlements should be partially inherited from the JDK you have installed.

Which Java version are you testing against?

chjan commented 3 years ago

I have not announced anything about bundling Java with my application

If you're using the system-wide Java, it actually simplifies this quite a bit; the entitlements should be partially inherited from the JDK you have installed.

Which Java version are you testing against?

No, this is incorrect. I have tried all 11 and 15 jdks. The sought after entitlement for microphone is not included. I ask again: Have you tried to run your test mic class as an executable jar? Yes or no?

tresf commented 3 years ago

I have tried all 11 and 15 jdks. The sought after entitlement for microphone is not included. I ask again: Have you tried to run your test mic class as an executable jar? Yes or no?

If you refuse to provide a specific JDK version, I'll assume AdoptOpenJDK11, latest from the website and test using that.

The sought after entitlement for microphone is not included

I've linked -- what I believe to be the correct entitlement -- above, which is included in the latest AdoptOpenJDK version, of which I'll be testing against.

tresf commented 3 years ago

Have you tried to run your test mic class as an executable jar

I've created an Application called SoundRecorder.app which is basically a stub to java -jar SoundRecorder.jar and no dialog is raised, a sound file is created where the WAV should be, but it contains no sound.

I haven't added entitlements (my experience says they shouldn't be needed unless distributing through the App Store -- since they're generally reserved only for sandboxed applications) but I'm not afraid to if needed.

I've found a lot of people complaining about this exact problem, but no concrete answers:

https://stackoverflow.com/questions/55727488/
"Application doesn't ask for permission to access microphone in MacOS 10.14 Mojave"
https://stackoverflow.com/questions/58728150
"How to give Microphone permission to Java application (macOS Mojave and above)?"
https://stackoverflow.com/questions/56084303
"OpenCV command line app can't access camera under macOS Mojave"
https://stackoverflow.com/a/66210338/3196753
"Cannot Access Microphone on Mac Mojave using pyaudio"
https://stackoverflow.com/questions/61989711
"How to record microphone on macos with pyaudio?"

... and a long thread about this exact same problem for VSCode users: https://github.com/microsoft/vscode/issues/95062 (still no solution). I believe PyCharm fixed it by requesting access from the macOS libraries directly, something that can be done with a 3rd-party Java tool, JNA.

I'll keep digging.

tresf commented 3 years ago

So it appears IntelliJ tackles this entitlement by baking it directly into the binary that launches the software. Inspecting the Contents/MacOS/idea binary, these are baked in:

Click to expand xml

```xml com.apple.security.cs.allow-jit com.apple.security.cs.allow-unsigned-executable-memory com.apple.security.cs.allow-dyld-environment-variables com.apple.security.cs.disable-library-validation com.apple.security.device.audio-input com.apple.security.device.camera com.apple.security.personal-information.location com.apple.security.personal-information.addressbook com.apple.security.personal-information.calendars com.apple.security.personal-information.photos-library com.apple.security.automation.apple-events ```

It's my theory that the codesign entitlements are needed directly on the binary itself, which seems to be a bit of a stretch when launching from a shell script. I will continue to investigate.

chjan commented 3 years ago

I have tried all 11 and 15 jdks. The sought after entitlement for microphone is not included. I ask again: Have you tried to run your test mic class as an executable jar? Yes or no?

If you refuse to provide a specific JDK version, I'll assume AdoptOpenJDK11, latest from the website and test using that.

The sought after entitlement for microphone is not included

I've linked -- what I believe to be the correct entitlement -- above, which is included in the latest AdoptOpenJDK version, of which I'll be testing against.

I have tried all 11 and 15 jdks. The sought after entitlement for microphone is not included. I ask again: Have you tried to run your test mic class as an executable jar? Yes or no?

If you refuse to provide a specific JDK version, I'll assume AdoptOpenJDK11, latest from the website and test using that.

The sought after entitlement for microphone is not included

I've linked -- what I believe to be the correct entitlement -- above, which is included in the latest AdoptOpenJDK version, of which I'll be testing against.

no, no, I do not refuse anything :-) . I am running 15.0.1 now, but the results are equal on other versions adoptopen, oracle, version 11 or 15, etcetc

chjan commented 3 years ago

I have tried all 11 and 15 jdks. The sought after entitlement for microphone is not included. I ask again: Have you tried to run your test mic class as an executable jar? Yes or no?

If you refuse to provide a specific JDK version, I'll assume AdoptOpenJDK11, latest from the website and test using that.

The sought after entitlement for microphone is not included

I've linked -- what I believe to be the correct entitlement -- above, which is included in the latest AdoptOpenJDK version, of which I'll be testing against.

I am pretty sure that THIS IS the correct entitlement, so please check that one in particular: https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_security_device_audio-input

chjan commented 3 years ago

Have you tried to run your test mic class as an executable jar

I've created an Application called SoundRecorder.app which is basically a stub to java -jar SoundRecorder.jar and no dialog is raised, a sound file is created where the WAV should be, but it contains no sound.

I haven't added entitlements (my experience says they shouldn't be needed unless distributing through the App Store -- since they're generally reserved only for sandboxed applications) but I'm not afraid to if needed.

I've found a lot of people complaining about this exact problem, but no concrete answers:

https://stackoverflow.com/questions/55727488/ "Application doesn't ask for permission to access microphone in MacOS 10.14 Mojave"

https://stackoverflow.com/questions/58728150"How to give Microphone permission to Java application (macOS Mojave and above)?"

https://stackoverflow.com/questions/56084303"OpenCV command line app can't access camera under macOS Mojave"

https://stackoverflow.com/a/66210338/3196753"Cannot Access Microphone on Mac Mojave using pyaudio"

https://stackoverflow.com/questions/61989711"How to record microphone on macos with pyaudio?"

... and a long thread about this exact same problem for VSCode users: microsoft/vscode#95062 (still no solution). I believe PyCharm fixed it by requesting access from the macOS libraries directly, something that can be done with a 3rd-party Java tool, JNA.

I'll keep digging.

Tres, please understand, I have given the correct and mandatory things that needs to be done with a stub as you call it (an javaapp.app folder containing a lot of subfolders). In short: Entitlements and signing. See above.

I dont understand what you mean when you say that entitlements are included in the jdk? Not at all. Entitlements are something you define ONLY WHEN you create your stub, through java packager or (hopefully) some much better alternative.

I say again: Try your test.jar executable by double clicking it in its folder. Predicted result: It will not get anything else but silence out of the input stream. Now try to run it in Apple's terminal app: Predicted result: MacOs BigSur will ask you "Do you want Terminal to be allowed to access mic?". You confirm, and then your app should work. In other words, when you start a .jar executable from Terminal, then Terminal is allowed to access mic. It is not Java that is granted this allowance then, it is the Terminal app. See?

So now, to create a commercial product for MacOs, we of course need to package the jar executable according to rules on MacOs. And this is to create an app "stub" as you call it.

Now we are heading for trouble because jpackager (java 14 and onwards) is a completely rotten piece of sw that will require its user to really know the details about what to write and not write in order to make it work. Then you need a Apple developer user to sign it, and there you go. This is the current situation, that this is EXTREMELY COMPLEX.

And therefore, the "java community" must liten to me when I state that it will be mandatory to refurbish jpackager into something much better, and with that I mean that jpackager should not be a terminal app, but a gui app that with a teaspoon describes the options the user has or must use. For example in my case, the audio input entitlement.

Am I getting through? I will kindly request that projectleaders of this thread step forward and make an intellgent remark about what to do. Because, clearly, someone must to something to improve this fatal situation. (Not that it is impossible to succeed, but you need to read both this and that blog to be successful. But "package for store" should NOT be difficult, it should be easy)

tresf commented 3 years ago

The entitlements are baked into the binary file itself. This post shows the binary file viewed in a hex editor:

https://github.com/AdoptOpenJDK/openjdk-support/issues/222#issuecomment-803639038 (click the link to expand the XML).

The apps that work seems to have this in common. More investigation to come.

aahlenst commented 3 years ago

Am I getting through?

No, because you're wrong here for that. If you want to see changes in OpenJDK, you have to engage with the OpenJDK project and this is not the OpenJDK project. The OpenJDK project is over at https://openjdk.java.net/.

chjan commented 3 years ago

OK, I accept that comment, but aren't we all engaged in ensuring that Java survives on Mac computers? Unfortunately, I am not in the position to engage in any jdk project, but I put some effort into this thread in order to create awareness of an acute problem. If you don't care, you don't care. I accept that as well. All is well.

aahlenst commented 3 years ago

I put some effort into this thread in order to create awareness of an acute problem.

If you want to create awareness or desire jpackage to be enhanced, you have to get OpenJDK involved because OpenJDK is the place where Java gets developed. We only build Java (and develop some installers) because OpenJDK is a source-only project.

tresf commented 3 years ago

aren't we all engaged in ensuring that Java survives on Mac computers?

This is much larger than Java. VSCode hasn't figured this out either and they do NOT use Java.

tresf commented 3 years ago

Repeating what was already mentioned in case it's gotten lost:

I've created a small, stub launcher which reproduces the original bug (a Java app won't ask for Microphone permissions).

I'll keep working from this example moving forward until I have a workable Application bundle which prompts for access. This will be a lot of trial and error until it works, some of which may involve invoking the Microphone APIs directly (which I believe is how PyCharm got around this issue).

I still see no evidence this is a Java-specific bug, but I do thanks @aahlenst for reopening as I think the finding will be valuable for macOS Desktop developers at large.

chjan commented 3 years ago

"We only build Java (and develop some installers) because OpenJDK is a source-only project"

Would you please point me to where in any readme file concerned with this project that this information is revealed? thanks.

Why do you spend time building something that you now know is not working on one of the major operating systems?

I shall move over to openjdk.java.net

tresf commented 3 years ago

@chjan I'm working hard to identify the source of this issue, which will take some time.

If that's insufficient, I'd happily work from a new issue which is not incumbered by off-topic content.

aahlenst commented 3 years ago

Would you please point me to where in any readme file concerned with this project that this information is revealed? thanks.

https://adoptopenjdk.net/about.html

chjan commented 3 years ago

@chjan I'm working hard to identify the source of this issue, which will take some time.

If that's insufficient, I'd happily work from a new issue which is not incumbered by off-topic content.

@tresf, I am monitoring the thread and will follow your progress! But the cause of the issue I have explained above.

chjan commented 3 years ago

Would you please point me to where in any readme file concerned with this project that this information is revealed? thanks.

https://adoptopenjdk.net/about.html

Thanks. Just a suggestion: How about putting this link into the intro of the github readme?

aahlenst commented 3 years ago

@chjan You mean https://github.com/AdoptOpenJDK/openjdk-support#issues-that-are-openjdk-or-openj9-upstream-issues? If so, I can improve the wording.

tresf commented 3 years ago

@tresf, I am monitoring the thread and will follow your progress! But the cause of the issue I have explained above

This statement is incorrect. There has been no cause identified above.

Why do you spend time building something that you now know is not working on one of the major operating systems?

Because it's a good way to simply illustrate the problem in the form of a unit test (one of which was requested several times but never provided) and thus, when the issue is resolved, will serve as passing criteria for closure.

but aren't we all engaged in ensuring that Java survives on Mac computers

This rhetoric has gone too far. My patience for this unprofessional interaction has grown too thin. With regret, I'm abandoning this bug report as well as all interactions with the OP. I will track this issue in a new, single, organized thread.

adoptium / adoptium-support