"Must match character set" message when logging in with 2FA enabled

[mbabker]

Steps to reproduce the issue

Log into a site after updating to 3.6.3.

Expected result

Successful login

Actual result

Error page with "Must match character set"

System information (as much as possible)

Hit this on www.joomla.org, downloads.joomla.org, developer.joomla.org, and my own website (Rochen & SiteGround hosting platforms, PHP 5.6 and 7.0).

Additional comments

On each of the four sites listed above, my accounts have 2FA enabled. I tested on another site where I do not have this feature enabled and successfully authenticated.

[mbabker]

From the logs:

Uncaught Exception of type Exception thrown. Stack trace:
#0 /libraries/fof/encrypt/base32.php(183): FOFEncryptBase32->toBin('')
#1 /libraries/fof/encrypt/totp.php(114): FOFEncryptBase32->decode(NULL)
#2 /plugins/twofactorauth/totp/totp.php(276): FOFEncryptTotp->getCode(NULL)
#3 [internal function]: PlgTwofactorauthTotp->onUserTwofactorAuthenticate(Array, Array)
#4 /libraries/joomla/event/event.php(69): call_user_func_array(Array, Array)
#5 /libraries/joomla/event/dispatcher.php(159): JEvent->update(Array)
#6 /libraries/joomla/application/base.php(106): JEventDispatcher->trigger('onUserTwofactor...', Array)
#7 /libraries/fof/integration/joomla/platform.php(545): JApplicationBase->triggerEvent('onUserTwofactor...', Array)
#8 /plugins/authentication/joomla/joomla.php(146): FOFIntegrationJoomlaPlatform->runPlugins('onUserTwofactor...', Array)
#9 /libraries/joomla/user/authentication.php(284): PlgAuthenticationJoomla->onUserAuthenticate(Array, Array, Object(JAuthenticationResponse))
#10 /libraries/cms/application/cms.php(833): JAuthentication->authenticate(Array, Array)
#11 /libraries/cms/application/administrator.php(324): JApplicationCms->login(Array, Array)
#12 /administrator/components/com_login/controller.php(64): JApplicationAdministrator->login(Array, Array)
#13 /libraries/legacy/controller/legacy.php(702): LoginController->login()
#14 /administrator/components/com_login/login.php(22): JControllerLegacy->execute('login')
#15 /libraries/cms/component/helper.php(405): require_once('')
#16 /libraries/cms/component/helper.php(380): JComponentHelper::executeComponent('')
#17 /libraries/cms/application/administrator.php(98): JComponentHelper::renderComponent('com_login')
#18 /libraries/cms/application/administrator.php(152): JApplicationAdministrator->dispatch()
#19 /libraries/cms/application/cms.php(261): JApplicationAdministrator->doExecute()
#20 /administrator/index.php(51): JApplicationCms->execute()
#21 {main}

@nikosdion The trace ends in FOF code. Can you look into this please?

[zero-24]

Confirmend maybe realted to the untested for update?

https://github.com/joomla/joomla-cms/pull/11673

[PhilETaylor]

fof/encrypt/totp.php ? paging @nikosdion :-)

[PhilETaylor]

I think this is a candidate for https://docs.joomla.org/Category:Version_3.6.3_FAQ

[brianteeman]

ITs a wiki - go for it

[mbabker]

You think!? I'm locked out of 3 .org properties! (Well, not completely locked out, but the fact I now have to go into the databases of at least 4 sites, remove my 2FA configuration, and reset it is annoying to say the least)

[PhilETaylor]

Like I have time to learn what a wiki is and how to use one :)

[zero-24]

https://docs.joomla.org/J3.x:Unable_to_login_with_2fa_enabled_after_3.6.3 please review

[Sandra97]

As usual, thank you @zero-24!

[PhilETaylor]

Im not sure how FoF can be called 3rd party software though... its more like core Joomla now :)

[Sandra97]

Make your suggestion of edits here and I'll apply them if you don't want to directly edit the wiki

[mbabker]

It is third party software distributed with the core Joomla package. Same with PHPMailer, or the lessc compiler.

[pe7er]

thanks @zero-24, I've added some category info to the bottom your wiki addition.

[zero-24]

Thanks @pe7er

This: https://github.com/joomla/joomla-cms/blob/staging/libraries/fof/autoloader/fof.php#L2-L7 should make clear it is third party software as Michael pointed out

[Sandra97]

I changed the sentence on JDocs using Michael's words

[PhilETaylor]

I actually think this problem is before the fof code, and in the onUserTwofactorAuthenticate method, the otp_config is not being loaded/passed in by Joomla for some reason... even further up the stack at https://github.com/joomla/joomla-cms/blob/a0b3e3ea5074dde2dab8252b3be394678905843b/plugins/authentication/joomla/joomla.php#L107 this might be the root issue.

At the moment Joomla doesnt pass any config to the fof code to decode!

[zero-24]

I have no yubikey here but i have two independet reports that yubikey works. So it looks like to be not a general problem with not passed configs?

[brianteeman]

Nope yubikey doesnt work - it completely ignores it and lets you login - at least thats what it did on my upgraded site Notice: Undefined index: yubikey in /plugins/twofactorauth/yubikey/yubikey.php on line 93

On 18 October 2016 at 19:49, zero-24 notifications@github.com wrote:

I have no yubikey here but i have two independet reports that yubikey works. So it looks like to be not a general problem with not passed configs?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/12458#issuecomment-254602651, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPH8b25lxbceYPyLWLSJRPrEyqaYs_qks5q1RSmgaJpZM4KaCxb .

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. https://brian.teeman.net/ http://brian.teeman.net/

[zero-24]

@mbabker something strange happen. Can you try it again? I did have removed the 2fa from my account and created a new one. but i can't reproduce the problem with them? Also if i don't enter a code or a wrong code i get the expected warning and i can't login? I'm more than confused?

[mbabker]

On the downloads site, I removed my otpKey and otep values from the database, deleted my old Google Authenticator config, and redid the setup for it. Logged out and able to log in again with 2FA.

[andrepereiradasilva]

tested a clean 3.6.3 and created a new user with 2FA with Google Authenticator and seems to work.

[zero-24]

So what happen? Do you have any site with 2FA enabled without removed values from the database?

[dandpclements]

I have the same problem - I restored the site using the backup taken as part of the upgrade and still have the same problem - I'm presuming the restore has been applied over the 3.6.3 release and therefore something new is left behind from 3.6.3?

[brianteeman]

@andrepereiradasilva can you try setting up TFA on a 3.6.2 site and then upgrading that

On 18 October 2016 at 20:15, dandpclements notifications@github.com wrote:

I have the same problem - I restored the site using the backup taken as part of the upgrade and still have the same problem - I'm presuming the restore has been applied over the 3.6.3 release and therefore something new is left behind from 3.6.3?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/12458#issuecomment-254609849, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPH8Qv0W_xFdVJsdplgBB6jK2-MYYdvks5q1RrYgaJpZM4KaCxb .

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. https://brian.teeman.net/ http://brian.teeman.net/

[mbabker]

@zero-24 I have 2FA enabled accounts on developer.joomla.org, www.joomla.org, and www.babdev.com I can still test with. But my account on downloads.joomla.org is most assuredly now completely reset and unusable to test any further with. I also have an account on seo.hdwebpros.com without 2FA but as we've isolated the issue to only the 2FA side of things that's irrelevant.

So what it seems is that something with the FOF upgrade has resulted in old data being unreadable. At least that's my first guess at a far off glance.

[wilsonge]

The root issue is we upgraded to a new version of FOF to deal with PHP 7.1 support https://github.com/joomla/joomla-cms/blob/staging/libraries/fof/encrypt/aes.php#L47

Result is that we were using FOFEncryptAesMcrypt and moved to FOFEncryptAesOpenssl - but this now means everyone on Mcrypts keys are invalid :(

[ggppdk]

Locked out of website is serious issue, and 2FA on administrator accounts of many websites is not uncommon.

Also what about other user accounts with 2FA ? they all have to do redo 2FA configuration ? how are they going to do this ?

Maybe the distribution of J3.6.3 should be stopped ?

I have just now renamed folder, plugins/twofactorauth/ so that users using 2FA (myself included) can login

[nikosdion]

As I said almost four years ago: encrypting the two factor authentication configuration with the site's secret key is a stupid, pointless idea. Yet the PLT had me implement it anyway, despite my objections, in order to accept the PR of the TFA feature.

The two factor authentication settings are stored in the users table. You had me encrypt them with the site's secret key. When you are trying to log in with TFA we read that data from the database and try to decrypt them using the current secret key. If there's a mismatch the decryption fails, the data is corrupt and you get this error message.

Another change made in FOF recently has to do with the way the encryption works. Up until Joomla 3.6.2 we would only use mcrypt. However, since mcrypt is now marked as dead, I have implemented decryption with OpenSSL as well and it's the preferred decryption. In all test environments I have the decryption works just fine either way. I suspect that if you are on a borked OpenSSL installation the decryption might break, throwing this error.

It all boils down to the PLT making a really wrong call despite my objections, forcing me to implement a USELESS encryption (since the key is stored in the same memory space as the encrypted data, duh!). Now excuse me, I'm in the middle of moving, packing stuff in boxes and my Internet connection will be cut off in 24 hours for a little over a week. Next time I tell you something is a bad idea just bloody trust me instead of waiting for 2 to 4 years until it blows up in your face and have me tell you "I told you so". Pretty please. We've been playing this game since 2009, it's becoming really tiresome.

[zero-24]

@wilsonge so we can workarrund the FOF B/C break by changing the order of the method tests? So if mycrypt is there use it over openSSL?

[mbabker]

@ggppdk Stopping distribution of the release fixes nothing and frankly (as someone who used to coordinate this stuff) I find it insulting when people suggest this after a release has been issued, especially given the open development process and the numerous calls for testing that go into releases. We are NOT talking about an issue that causes the site to fail over miserably. Ya, you can't log in now, but is your site experiencing fatal errors? That's the only condition I find acceptable to stop distribution.

[ggppdk]

I find it insulting when people suggest this after a release has been issued, especially given the open development process and the numerous calls for testing that go into releases. We are NOT talking about an issue that causes the site to fail over miserably. Ya, you can't log in now, but is your site experiencing fatal errors? That's the only condition I find acceptable to stop distribution.

I am sorry, i was just asking

[mbabker]

It's not something specifically against you, but as someone who did this stuff and dealt with people crying "my site is broken, Joomla sucks, stop distributing the package", it gets annoying having to keep hearing it. Even when I haven't been involved in that process for close to two years.

[mbabker]

I suspect that if you are on a borked OpenSSL installation the decryption might break, throwing this error

Well, glad to know that Rochen and SiteGround have borked OpenSSL installations...

[andrepereiradasilva]

@brianteeman yes, can confirm, 2FA in clean 3.6.2 work fine. upgrade to 3.6.3, login with 2FA we get the error. aditional info: tested in CentOS 7.x (latest) with centos openssl package

[brianteeman]

Your comment and that aof andre actually confirms that it is

On 18 October 2016 at 20:38, Alexander Jochum notifications@github.com wrote:

Currently debugging the code and at the moment to me it looks like decoding the old TwoFactorAuthentication config doesn't work anymore in getOtpConfig of com_user/models/users.php file.

$otpConfig->config = @json_decode($decryptedConfig);

will return null and so everything afterwards fails.

Disabling 2FA temporary to login, disabling 2FA and reenabling 2FA again worked for me. In my opinion the problem is not the third party library for 2FA..

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/12458#issuecomment-254615911, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPH8XQeXShBApjGEoHrT0XusiWhKzdBks5q1SAfgaJpZM4KaCxb .

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. https://brian.teeman.net/ http://brian.teeman.net/

[DarkCloud14]

@brianteeman Sorry my fault I missed the line $aes = new FOFEncryptAes($key, 256);

after replacing the fof library folder of Joomla 3.6.3 installation with one of Joomla 3.6.2 it worked again. That's why I removed my comment again, I was too fast. Sorry

[zero-24]

http://stackoverflow.com/questions/31520311/decrypt-mcrypt-with-openssl

If you encrypt in mcrypt without adding PKCS7 manually, mcrypt will happily pad your plaintext with NUL bytes. OpenSSL will do PKCS7 padding for you whenever using aes-X-cbc. The unfortunate consequence of this is that if you have AES-CBC(NULL_PADDED(plaintext)) and try to decrypt it, openssl_decrypt will attempt to remove the padding and fail.

Maybe this can help more high level PHP developers than me, do we do anything with PKCS7 and mycrypt i can't find anything?

If not it sounds to me that is is not a borked (whatever borked means) OpenSSL installation

[wilsonge]

https://github.com/joomla/joomla-cms/pull/12461 This will fix things as a VERY temporary measure. But Tobias you have the right idea here. We somehow need openssl to verify these keys if at all possible. I'm going to try and work on this later

[Hutchy68]

1. Log in, get error, click return to control panel which brings you back to log in.
2. Log in without putting in your secret key and you are now logged in. 👍 The bad part is key is still living so if you log out you can log back in without the secret key!!! 👎
3. Then you can disable 2FA, then re-enable with a new key. 2FA starts working as expected again.

[brianteeman]

Thats exctly what I got on my test site

On 18 October 2016 at 21:12, Tom Hutchison notifications@github.com wrote:

1.

Log in, get error, click return to control panel which brings you back to log in. 2.

Log in without putting in your secret key and you are now logged in. 👍 The bad part is a session is still living so if you log out you can log back in without the secret key!!! 👎 3.

Then you can disable 2FA, then re-enable with a new key. 2FA starts working as expected again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/joomla/joomla-cms/issues/12458#issuecomment-254625245, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPH8eyFtgPHunWhIYqLdeHk_6MzU29sks5q1Sg6gaJpZM4KaCxb .

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. https://brian.teeman.net/ http://brian.teeman.net/

[DarkCloud14]

Not sure if this is really a fixable bug without some conversion if Joomla wants to use openssl instead of mcrypt in the future.

What I tried was to add a class which extends FOFUtilsPhpfunc and override the __call function to return false.

class TwoFactorAuthConversion extends FOFUtilsPhpfunc { public function __call($func, $args) { return false; } }

Now every creation of an FOFEncryptAes object must be modified so that we pass a new object of our new class as last parameter.

In administrator/components/com_user/models/users.php for example every

new FOFEncryptAes($key, 256);

must be changed to

new FOFEncryptAes($key, 256, 'cbc', new TwoFactorAuthConversion() );

this will then force FOFEncryptAes to use FOFEncryptAesMcrypt class as adapter instead of the openssl class. It's not really nice but this wouldn't require a modification of the FOF library.

Maybe the updater could do something like this and convert the existing OTP configurations of each user from mcrypt to openssl.

This will of course not help any user who already updated to Joomla 3.6.3..

[DarkCloud14]

Ok I added an automated conversion in getOtpConfig function. You must of course add the TwoFactorAuthConversion class as stated in my previous post (only the class not the other modifications!). But then this will automatically tries the mcrypt method for decryption and if successful calls setOtpConfig to set the configuration in OpenSSL format..

Here is the code that I modified in getOtpConfig of administrator/components/com_user/models/users.php file.

        /*
         * If the decryption failed for any reason we essentially disable the
         * two-factor authentication. This prevents impossible to log in sites
         * if the site admin changes the site secret for any reason.
         */
        if (is_null($otpConfig->config))
        {
            $mcryptConversion = new FOFEncryptAes($key, 256, 'cbc', new TwoFactorAuthConversion());

            // Decrypt the data
            $decryptedConfig = $mcryptConversion->decryptString($encryptedConfig);
            $decryptedOtep = $mcryptConversion->decryptString($encryptedOtep);

            // Remove the null padding added during encryption
            $decryptedConfig = rtrim($decryptedConfig, "\0");
            $decryptedOtep = rtrim($decryptedOtep, "\0");

            // Update the configuration object
            $otpConfig->method = $method;
            $otpConfig->config = @json_decode($decryptedConfig);
            $otpConfig->otep = @json_decode($decryptedOtep);

            // If it worked now and config isn't null we call setOtpConfig to do the conversion to OpenSSL else behave as before.
            if (is_null($otpConfig->config))
            {
                $otpConfig->config = array();
            }
            else
            {
                $this->setOtpConfig($user_id, $otpConfig);
            }
        }

[nikosdion]

God, no! That TwoFactorAuthConversion crap makes all calls to PHP builtins to return false without executing! That screws up the decryption, essentially always returning a null reply. That's the most roundabout way you can go to disabling TFA. Better just delete the TFA plugins altogether, then!

Apparently none of you is capable of seeing the simple truth right in front of your noses. I am disappointed. Doubly for seeing the crazy workarounds which only compound the problem instead of solving it. Stop breaking Joomla, damn you!!! THINK BEFORE YOU CODE!!!

A cursory examination of FOFEncryptAesOpenssl::setEncryptionMode reveals the nature of the problem and why I'm talking about broken OpenSSL implementations. When you instantiate an AES 256-bit key CBC instance the adapter sets the target OpenSSL algorithm to aes-256-cbc. However, if the OpenSSL extension does not report that algorithm as available the $defaultAlgo is used instead. On these broken implementation you have aes-128-cbc but not aes-256-cbc. Since the wrong algorithm is used you end up with garbage decoded data which cannot be parsed as JSON, throwing the error message you get.

The only thing you could do is forcibly instantiate the FOFEncryptAesMcrypt adapter and decode the data using the same code as FOFEncryptAes::decryptString (the other methods you need are public on that class).

This is a short term solution which will throw warnings in PHP 7.1 and probably break in 7.2. Ideally you'd use the trick above to decrypt the data and then write it decrypted to the database. I could analyze the threat models and show you that encryption makes no sense for TFA configuration but I've already done that four years ago and the PLT seems to have been too dumb to even pretend to listen to me. Meh.

Now please allow me to go to sleep because I'm still in the middle of packing an entire house into boxes for moving.

[wilsonge]

My algo is aes-256-cbc in that method but I'm still experiencing the break. But as you describe the Json data is clearly garbage coming out. Still trying to investigate where exactly the problem lies and if it's something to do with the white padding of the mcrypt message as tobias' stack overflow link implies or something unrelated

[wilsonge]

OK So I'm sure that I've got the aes-256-cbc locally and @mbabker tested and got that same result on Rochen. So at least on some hosts there appears to be a deeper problem. I looked into the issue @zero-24 supplied on StackOverflow and whilst adding PKCS7 to the mcrypt appeared to make some sort of difference (it removed white space from the result) but it certainly didn't fix the issue - the decrypted data still was meaningless - just without whitespace in it.

I've set myself up a basic script of 4 files that encrypts in the FOFEncryptAes from J3.6.2 then decrypts it with FOFEncryptAes from 3.6.3 (with some hacking to remove some of the new 3.6.3 dependencies which were not required).

Will take another shot at things tomorrow. But have to say I'm running out of ideas fast :(

[DarkCloud14]

@nikosdion That's true but I'm pretty sure that I never said that this is a good or the final solution and I also never created a pull request for this as there are other things I don't like about this.

I just posted it that someone else with a bit more brain could find a better solution based on this, without modifying the original third party library and writing its own decryption method using FOFEncryptAesMcrypt.

This also wouldn't break TFA completely as the TwoFactorAuthConversion object is only used in FOFEncryptAes constructor to do some checks and if they fail it'll use mcrypt instead of openssl.

To not return false in general you could modify TwoFactorAuthConversion __call function so that it behaves as usual for everything but return false for a function call done in FOFs FOFEncryptAesOpenssl isSupported function, like $phpfunc->function_exists('openssl_get_cipher_methods')

Beside the fact that my class naming is also pretty shitty I also wouldn't add this piece of crap code in that way into standard Joomla code as final solution. Also I would only add somekind of conversion functionality (not my crappy solution) in Joomlas upgrade process and not its standard code.

In my opinion this problem shouldn't have found its way into the final release at all. Maybe it would have been better to use the previous version of FOF library until there was a good solution for this problem but hey I'm just someone who writes shitty code so what do I know.

[nikosdion]

@wilsonge Don't just revert FOF to the previous version and release Joomla again. As we already established, FOF 2 was not PHP 7 compatible.

I insist that keeping the useless encryption of the TFA config makes no sense. Do you want me to spell it out for you? If I can run arbitrary PHP or SQL code on your site (the only ways to read or affect the config data in the db) I can just CHANGE your password and REMOVE TFA with a simple SQL statement. Encryption makes no sense whatsoever. I have told you so 4 years ago. Will you FINALLY listen to me?

How to do it. Read the config. Split it on : as we do now which leaves you with the plugin name and the config data. Check if there's a { in the config data and try to parse it as JSON (the config data is always an object). If the former is false or the latter yields null the data is encrypted. Decrypt it and retry. Upon success save it back to the db. It's not rocket science.

Finally, let me please note that FOF 2 is END OF LIFE. I'm done trying to maintain it for Joomla 3. I tried really hard to backport necessary changes to support PHP versions that FOF 2 was never intended to support but there's only so much I can do and nobody seems to give a flying damn about doing proper testing even when I explicitly ask you to. I don't see the point of assuming all risk of a release and having zero control over it so from now on you're strictly on your own.

[ggppdk]

Then when about when upgrading from J3.6.2 , reconfiguration of TFA is forced my Joomla ?

I mean if upgrading from <=J3.6.2, then existing data are cleared, and a new message appears in backend dashboard about the fact that data had been cleared, and need to reconfigure it

[mbabker]

In my opinion this problem shouldn't have found its way into the final release at all.

You're right. It shouldn't have. The fact it did means that not enough folks tested either the pull request updating FOF (in August) or after its merge updating sites to the current development head with 2FA enabled. There's no point in further beating the dead horse, stuff happens.

Then when about when upgrading from J3.6.2 , reconfiguration of TFA is forced my Joomla ?

Unacceptable.

[PhilETaylor]

Can we start a "joomla RC testing checklist" then? (said it before... said it again) so that we can add things like this to a solid checklist so that this "feature" is always tested by testers ?