Error During LDAP Setup

[C0d3rZ3r0]

There seems to be a bug with the way LDAP environment variables are translated into config.php variables.

It gives me this this error when I uncomment the LDAP block in .env file and provide the config values:

...
MISP | Set Up LDAP ...
Provided new config is not an array, `NULL` given.
...

At least one of the LDAP values is not being transformed into a valid PHP array. This needs to be fixed.

It would also help if those errors could be made more descriptive & correlatable with variable names in the .env file.

[UFOSmuggler]

G'day mate,

I just tried this and it worked fine. Can you please confirm your values for the following variables are arrays?

LDAP_FILTER LDAP_EMAIL_FIELD

I can get the same error if for example I specify LDAP_EMAIL_FIELD="mail" instead of perhaps LDAP_EMAIL_FIELD='["mail"]' or similar.

[C0d3rZ3r0]

I believe you will need to escape the quotes. I'm using the default values from the template:

...
 LDAP_FILTER="[\"mail\", \"uid\", \"cn\" ]"
 LDAP_DEFAULT_ROLE_ID="3"
 LDAP_DEFAULT_ORG="1"
 LDAP_EMAIL_FIELD="[\"mail\"]"
...

I've only removed the comment hash prefixed to each line in the LDAP block. I believe the field names are the same for my LDAP instance, so no change is necessary here.

[UFOSmuggler]

I didn't need to escape the double quotes in my example because they were within a single quote, not a double quote. Escaped double quotes within double quotes also worked for me though. I too also initially just removed the comment hashes to test. It worked just fine.

Can you please try again and provide the entire LDAP envar config if it fails again?

Also please tell me which git commit your repo is at, and which misp-core docker image you have.

[C0d3rZ3r0]

I'm on the latest version & commit on the main branch. Version:

##
# Build-time variables
##

CORE_TAG=v2.4.193
MODULES_TAG=v2.4.193
PHP_VER=20190902
LIBFAUP_COMMIT=3a26d0a

Commit:

commit 303ea9d2f9e02735f9dae0e0866c0520ccc102f1 (HEAD -> master)
Author: Stefano Ortolani <ortolanis@vmware.com>
Date:   Wed Jun 12 15:28:38 2024 +0100

    Revert some changes to unblock new users

I've tried again on Windows and on Linux with the exact same config, and it gave me the exact same error "Provided new config is not array...". I noticed that I had not deleted the preceding space (after the '#'), so I tried deleting that too, but it returned the same error. Then I removed escaping from double quotes and enclosed both arrays in single quotes like you suggest, but sans luck.

Here is a redacted envvar block for LDAP config from my .env file:

LDAP_ENABLE=true
LDAP_APACHE_ENV="REMOTE_USER"
LDAP_SERVER="ldaps://ds.test.local" # actual URL redacted
LDAP_STARTTLS=true
LDAP_READER_USER="uid=service_acct,ou=service_acct_group,dc=example,dc=com" # actual details redacted
LDAP_READER_PASSWORD="complex_password_here" # password redacted
LDAP_DN="cn=qwerty_qwerty,ou=qwerty_qwerty,dc=example,dc=com" # actual details redacted
LDAP_SEARCH_FILTER=""
LDAP_SEARCH_ATTRIBUTE="uid"
LDAP_FILTER="[\"mail\", \"uid\", \"cn\" ]"
LDAP_DEFAULT_ROLE_ID="3"
LDAP_DEFAULT_ORG="1"
LDAP_EMAIL_FIELD="[\"mail\"]"
LDAP_OPT_PROTOCOL_VERSION="3"
LDAP_OPT_NETWORK_TIMEOUT="-1"
LDAP_OPT_REFERRALS=false

Were you presented with the LDAP authentication prompt when you tried? It's supposed to be a basic auth pop-up box as mentioned in https://github.com/MISP/MISP/issues/6189 . I do not see any such pop box, owing to the LDAP setup error.

[UFOSmuggler]

You're several commits behind, but i'll switch to your commit and see if I get the same thing.

[UFOSmuggler]

I'm unsure what the issue is man, here's what I get:

$ git status
HEAD detached at 303ea9d

$ docker image ls
REPOSITORY                              TAG       IMAGE ID       CREATED       SIZE
ghcr.io/misp/misp-docker/misp-core      latest    1da17e0afc68   5 days ago    1.01GB
ghcr.io/misp/misp-docker/misp-modules   latest    f05eae251d3b   5 days ago    1.15GB
ixdotai/smtp                            latest    e7334421ba0b   6 days ago    93.6MB
mariadb                                 10.11     6aa03f63fdf6   2 weeks ago   404MB
valkey/valkey                           7.2       496b4fc535c3   4 weeks ago   138MB

.env:

LDAP_ENABLE=true
LDAP_APACHE_ENV="REMOTE_USER"
LDAP_SERVER="ldaps://ds.test.local" # actual URL redacted
LDAP_STARTTLS=true
LDAP_READER_USER="uid=service_acct,ou=service_acct_group,dc=example,dc=com" # actual details redacted
LDAP_READER_PASSWORD="complex_password_here" # password redacted
LDAP_DN="cn=qwerty_qwerty,ou=qwerty_qwerty,dc=example,dc=com" # actual details redacted
LDAP_SEARCH_FILTER=""
LDAP_SEARCH_ATTRIBUTE="uid"
LDAP_FILTER="[\"mail\", \"uid\", \"cn\" ]"
LDAP_DEFAULT_ROLE_ID="3"
LDAP_DEFAULT_ORG="1"
LDAP_EMAIL_FIELD="[\"mail\"]"
LDAP_OPT_PROTOCOL_VERSION="3"
LDAP_OPT_NETWORK_TIMEOUT="-1"
LDAP_OPT_REFERRALS=false

docker compose logs -f misp-core:

...snip...
misp-core-1  | 2024-06-25T08:22:57.029239461Z Successfully updated 344 object templates.
misp-core-1  | 2024-06-25T08:22:57.035036771Z MISP | Set Up OIDC ...
misp-core-1  | 2024-06-25T08:22:57.035088201Z ... OIDC authentication disabled
misp-core-1  | 2024-06-25T08:22:57.035096341Z MISP | Set Up LDAP ...
misp-core-1  | 2024-06-25T08:22:57.134580240Z MISP | Set Up AAD ...
misp-core-1  | 2024-06-25T08:22:57.134601360Z ... Entra (AzureAD) authentication disabled
...snip...

config.php:

  'ApacheSecureAuth' => 
  array (
    'apacheEnv' => 'REMOTE_USER',
    'ldapServer' => 'ldaps://ds.test.local',
    'starttls' => true,
    'ldapProtocol' => 3,
    'ldapNetworkTimeout' => -1,
    'ldapReaderUser' => 'uid=service_acct,ou=service_acct_group,dc=example,dc=com',
    'ldapReaderPassword' => 'complex_password_here',
    'ldapDN' => 'cn=qwerty_qwerty,ou=qwerty_qwerty,dc=example,dc=com',
    'ldapSearchFilter' => '',
    'ldapSearchAttribut' => 'uid',
    'ldapFilter' => 
    array (
      0 => 'mail',
      1 => 'uid',
      2 => 'cn',
    ),
    'ldapDefaultRoleId' => 3,
    'ldapDefaultOrg' => '1',
    'ldapAllowReferrals' => false,
    'ldapEmailField' => 
    array (
      0 => 'mail',
    ),
  ),

i'm blowing everything away every time so this is all fresh containers, fresh volume mounts, fresh mysql volume etc....

can you do a git diff and see if you have accidentally changed anything? particularly the LDAP stuff in docker-compose.yml? i'm wondering if maybe you accidentally filled stuff in there instead of just .env?

are you using the pre-built images, or are you building your own?

is there anything you can think of that might help me get the same error?

[UFOSmuggler]

As an aside, I wonder if ldapSearchAttribut is a typo? It wouldn't cause this error, but maybe this is also a mistake. I will check this out tomorrow.

https://github.com/MISP/misp-docker/blob/master/core/files/configure_misp.sh#L144

[C0d3rZ3r0]

The PHP variable "ldapSearchAttribut" should be without an 'e'. That is the correct internal variable name, as per https://github.com/MISP/MISP/issues/6189 .

I think I posted the git log output from the wrong host. Here's the output from my Linux host where it was tested. I see that this is the current commit on the main branch on GitHub

commit 5d641c75e4d8f0d68579cc3450b193e7202f8d1b
Author: Stefano Ortolani <ortolanis@vmware.com>
Date:   Thu Jun 20 09:01:30 2024 +0100

    Fix template

[C0d3rZ3r0]

Additionally, I also tried changing the line endings for the .env file from CRLF (as on Windows) to LF (as on *nix). This still didn't help.

[C0d3rZ3r0]

i'm blowing everything away every time so this is all fresh containers, fresh volume mounts, fresh mysql volume etc....

Ditto.

can you do a git diff and see if you have accidentally changed anything? particularly the LDAP stuff in docker-compose.yml? i'm wondering if maybe you accidentally filled stuff in there instead of just .env?

I just copied over the template.env into .env. The only change in the docker-compose.yml has been the host port numbers - I mapped misp-core's 80/443 into my hosts 8080/8443 for testing. I'm certain I changed nothing else, and git diff confirms this.

are you using the pre-built images, or are you building your own?

I'm using the pre-built images.

is there anything you can think of that might help me get the same error?

No. 🙂

[ostefano]

Try removing that configs directory in-between deployments.

[UFOSmuggler]

Digging deeper into what is going on here.

It looks like your issue basically stems from an incorrectly formed json object being passed to the php json_decode() function. for instance the "ldapEmailField"(https://github.com/MISP/misp-docker/blob/master/core/files/configure_misp.sh#L149) object key needs to contain some kind of valid json type with a valid value. if we supply mail on its own, outside of quotes, it will shit the bed. similarly if we supply [mail] this is a valid json type with an invalid value within, this will also shit, but ["mail"] will be fine.

when the modify_config.php file is called, it tries to decode its input as json using PHP's json_decode() function, which if it fails it returns NULL.

the error message you receive suggests you received NULL, and is generated directly after testing to see if it has been transformed from json into a php associative array.

the json object is formed in configure_misp.sh here from the environment variables provided.

given this, we have to expect that something in one of the envars is causing this to not be valid json.

could you please shell into the container this fails on (probably docker exec -it misp-docker-misp-core-1 /bin/bash) and give me a redacted version of what this returns:

echo "{                       
        \"ApacheSecureAuth\": {
            \"apacheEnv\": \"${LDAP_APACHE_ENV}\",
            \"ldapServer\": \"${LDAP_SERVER}\",
            \"starttls\": ${LDAP_STARTTLS},
            \"ldapProtocol\": ${LDAP_OPT_PROTOCOL_VERSION},
            \"ldapNetworkTimeout\": ${LDAP_OPT_NETWORK_TIMEOUT},
            \"ldapReaderUser\": \"${LDAP_READER_USER}\",
            \"ldapReaderPassword\": \"${LDAP_READER_PASSWORD}\",
            \"ldapDN\": \"${LDAP_DN}\",
            \"ldapSearchFilter\": \"${LDAP_SEARCH_FILTER}\",
            \"ldapSearchAttribut\": \"${LDAP_SEARCH_ATTRIBUTE}\",
            \"ldapFilter\": ${LDAP_FILTER},
            \"ldapDefaultRoleId\": ${LDAP_DEFAULT_ROLE_ID},
            \"ldapDefaultOrg\": \"${LDAP_DEFAULT_ORG}\",
            \"ldapAllowReferrals\": ${LDAP_OPT_REFERRALS},
            \"ldapEmailField\": ${LDAP_EMAIL_FIELD}
        }
    }"

you could also try piping it to jq . as well and seeing if it validates.

this information should make it readily apparent where the issue lies.

[C0d3rZ3r0]

This is the output of the command:

{                       
        "ApacheSecureAuth": {
            "apacheEnv": "REMOTE_USER",
            "ldapServer": "ldaps://ds.test.local",
            "starttls": true,
            "ldapProtocol": 3,
            "ldapNetworkTimeout": -1,
            "ldapReaderUser": "uid=misp_service_acct,ou=service_acct_group,dc=corp,dc=example,dc=com",
            "ldapReaderPassword": "complex_password_here",
            "ldapDN": "cn=qwerty_qwerty,ou=qwerty_qwerty,dc=corp,dc=example,dc=com",
            "ldapSearchFilter": "",
            "ldapSearchAttribut": "uid",
            "ldapFilter": ["mail", "uid", "cn" ],
            "ldapDefaultRoleId": 3,
            "ldapDefaultOrg": "1",
            "ldapAllowReferrals": false,
            "ldapEmailField": ["mail"]
        }
    }

When I piped it to jq ., it said

parse error: Invalid escape at line 9, column 68

This is the line with the password. There's a slash in my real password that needed escaping.

This was way too silly.

I tried escaping it once ('\' to '\'), but that didn't work, so I had to escape it twice to bring it in the correct final format.

We need to add a comment on the password line reminding users to DOUBLY escape all necessary special characters in their password.

For instance, if the original password string is "pass\word" with a '\' in the middle, then the final escaped form (in the env variable on the container) needs to be exactly "pass\word" with '\' in the middle. To do this, the password in the .env config file needs to be written as "pass\\word", with doubly escaped '\', one for docker compose, one for the app.

Once an appropriate note is added against the password field, in template.env, we should close this ticket.

This is not a bug.

[ostefano]

Cool find! Could you open a PR with the proposed documentation fix?

Thanks!

[UFOSmuggler]

great result mate, thank you for persisting and helping us. a pr to the documentation would be awesome if you could!

[C0d3rZ3r0]

https://github.com/MISP/misp-docker/pull/84