search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.13k stars 5.47k forks source link

Postgres cluster is always created with SQL_ASCII encoding, regardless of locale #4543

Closed bkonkle closed 10 years ago

bkonkle commented 11 years ago

In our state, we're making sure to set the locale to UTF-8 before installing Postgres. If the locale is not set, Postgres defaults to SQL_ASCII, causing problems with our applications. Our state looks like this:

en_US.UTF-8:
    locale.system

postgresql:
    pkg.installed:
        - names:
            - postgresql
            - postgresql-contrib
        - require:
            - locale: "en_US.UTF-8"

Unfortunately, regardless of the locale the cluster is always generated at install with the SQL_ASCII encoding. If we drop and recreate the cluster, however, it uses the correct UTF-8 encoding.

We've tried running update-locale manually and restarting the salt-minion before running state.highstate to make sure that the process has the right environment, but it still initializes the cluster with SQL_ASCII. We've also tried adding the locale environment variables to salt-minion's upstart script. The only thing that works is installing Postgres manually outside of Salt.

This happens for us when using Salt with Rackspace, but I'm not entirely certain it's isolated to just Rackspace.

jsuchal commented 10 years ago

@fivethreeo no, that's just setting default value, when it's not set otherwise. http://www.tutorialspoint.com/python/dictionary_setdefault.htm

fivethreeo commented 10 years ago

Exactly, but we set that in - env :)

On 14 April 2014 22:26, Jano Suchal notifications@github.com wrote:

@fivethreeo https://github.com/fivethreeo no, that's just setting default value, when it's not set otherwise. http://www.tutorialspoint.com/python/dictionary_setdefault.htm

— Reply to this email directly or view it on GitHubhttps://github.com/saltstack/salt/issues/4543#issuecomment-40413671 .

Øyvind Saltvik

fivethreeo commented 10 years ago

Finally got a working solution :)

https://gist.github.com/fivethreeo/11046181

locale-gen en_US.UTF-8: # Locale must exist
  cmd.run:
    - unless: locale -a | grep -q en_US.utf8

en_US.UTF-8:
  locale.system:
    - require:
      - cmd: locale-gen en_US.UTF-8

postgresql:
  pkg.installed:
    - names:
      - postgresql-9.1
      - postgresql-contrib-9.1
    - env:
        LC_ALL: en_US.UTF-8 # NOTE! No - and two more spaces here.
    - require:
      - locale: en_US.UTF-8 

/etc/postgresql/9.1/main:
    file.directory:
    - user: postgres
    - group: postgres
    - dir_mode: 755
    - makedirs: true
    - require:
      - pkg: postgresql

postgres-hba-conf:
  file.managed:
    - name: /etc/postgresql/9.1/main/pg_hba.conf
    - source: salt://postgresql/pg_hba.conf
    - user: postgres
    - group: postgres
    - require:
      - file: /etc/postgresql/9.1/main

postgres-conf:
  file.managed:
    - name: /etc/postgresql/9.1/main/postgresql.conf
    - source: salt://postgresql/postgresql.conf
    - group: postgres
    - require:
      - file: /etc/postgresql/9.1/main

postgresql-service:
  service.running:
    - name: postgresql
    - enable: True
    - require:
      - pkg: postgresql
    - watch:
      - file: postgres-conf
      - file: postgres-hba-conf
fivethreeo commented 10 years ago

Consider closing this one and adding tickets for:

Preferably: Allow env to be a yaml list of key, value pairs.

or Clarify use of env as a true yaml dict in cmd.run and pkg states.

Add a locale-present.system state, and add - present to locale.system.

fivethreeo commented 10 years ago

Env issue is fixed in develop.

A locale.present state, and a - present to locale.system would be useful.

fivethreeo commented 10 years ago

In the develop version of salt this should now work:

en_US.UTF-8:
    locale.present

postgresql:
    pkg.installed:
        - names:
            - postgresql
            - postgresql-contrib
        - require:
            - locale: en_US.UTF-8
        - env:
            - LC_ALL: en_US.UTF-8
bdejong commented 10 years ago

If you already set the locale with locale.present, do you then still need to pass the env?

fivethreeo commented 10 years ago

Present as in existing on the system at all, we set env so LC_ALL is not set to C.

On 25 April 2014 08:56, bdejong notifications@github.com wrote:

If you already set the locale with locale.present, do you then still need to pass the env?

— Reply to this email directly or view it on GitHubhttps://github.com/saltstack/salt/issues/4543#issuecomment-41363859 .

Øyvind Saltvik

techdragon commented 10 years ago

@fivethreeo thanks for the persistence and clarity on this, I was very perplexed when I rebuilt a DB the other day and noticed this.

thedrow commented 10 years ago

So, this issue is finally resolved?

bmcorser commented 10 years ago

This is not working for me ...

I'm running an up-to-date version of Salt:

Master:

root@salt-master:/home/vagrant# salt --version
salt 2014.1.0-6638-g2a0a52c (Hydrogen)

Minion:

root@salt-minion:/home/vagrant# salt-call --version
salt-call 2014.1.0-6638-g2a0a52c (Hydrogen)

Here is my postgres.sls file:

en_GB.UTF-8:
  locale.present

postgres-pkgs:
  pkg.installed:
    - names:
      - postgresql
      - postgresql-server-dev-9.3
      - postgresql-client
    - require:
      - locale: en_GB.UTF-8
    - env:
      - LC_ALL: en_GB.UTF-8

postgresql:
  service.running:
    - require:
      - pkg: postgres-pkgs

postgres-apt:
  pkgrepo.managed:
    - name: "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main"
    - file: /etc/apt/sources.list.d/pgdg.list
    - key_url: http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc
    - require_in:
        - pkg: postgres-pkgs

However, after running salt-call state.sls postgres on my minion, the Postgres tables end up being encoded to LATIN1:

postgres@salt-minion:/home/vagrant$ psql -c '\l'
                             List of databases
   Name    |  Owner   | Encoding | Collate | Ctype |   Access privileges   
-----------+----------+----------+---------+-------+-----------------------
 postgres  | postgres | LATIN1   | en_US   | en_US | 
 template0 | postgres | LATIN1   | en_US   | en_US | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
 template1 | postgres | LATIN1   | en_US   | en_US | =c/postgres          +
           |          |          |         |       | postgres=CTc/postgres
(3 rows)
jsuchal commented 10 years ago

@bmcorser I think you still miss the - reset_system_locale: false

bmcorser commented 10 years ago

Same issue even if I use that.

fivethreeo commented 10 years ago

Try

postgres-pkgs:
  pkg.installed:
    - names:
      - postgresql
      - postgresql-server-dev-9.3
      - postgresql-client
    - require:
      - locale: en_GB.UTF-8
    - env:
         LC_ALL: en_GB.UTF-8
bmcorser commented 10 years ago

I am using version 2014.1.0-6638-g2a0a52c (Hydrogen) and this is still failing for me.

Running salt-call -l debug state.sls postgres on the minion succeeds, but my Postgres database still bears en_US encoding.

The locale state appears to succeed:

[INFO    ] Completed state [en_GB.UTF-8] at time 10:13:01.896501

However, the env doesn't seem to catch the apt-get command:

[INFO    ] Executing command ['apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'postgresql-server-dev-9.3'] in directory '/root'

I don't see any mention of LC_ALL in that command (assuming there should be).

fivethreeo commented 10 years ago

But if it did not work you should get ASCII, are you setting default locale anywhere in your states?

number5 commented 10 years ago

On Wed, May 28, 2014 at 8:19 PM, B M Corser notifications@github.comwrote:

en_GB.UTF-8

Has en_GB.UTF-8 been generated by locale-gen? What's the output of locale before and after the salt-call -l debug state.sls postgres?

simple is good http://brucewang.net http://twitter.com/number5

bmcorser commented 10 years ago

@fivethreeo I'm only setting locale in the state I posted.

@number5, on a fresh minion, I see:

$ locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_US"
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=en_US

So I get LANG but nothing else. After running salt-call state.sls postgres I see the same thing:

# locale
LANG=en_GB.UTF-8
LANGUAGE=
LC_CTYPE="en_US"
LC_NUMERIC="en_US"
LC_TIME="en_US"
LC_COLLATE="en_US"
LC_MONETARY="en_US"
LC_MESSAGES="en_US"
LC_PAPER="en_US"
LC_NAME="en_US"
LC_ADDRESS="en_US"
LC_TELEPHONE="en_US"
LC_MEASUREMENT="en_US"
LC_IDENTIFICATION="en_US"
LC_ALL=en_US

I'm thinking is locale.present enough? Should I set the locale to en_GB.UTF-8 as well?

fivethreeo commented 10 years ago

locale.present is just to make sure the locale exists

LC_ALL ENV to make the deb use the right locale

what bash shows is of no use, use cmd.run locale

On 28 May 2014 22:20, B M Corser notifications@github.com wrote:

@fivethreeo https://github.com/fivethreeo I'm only setting locale in the state I posted.

@number5 https://github.com/number5, on a fresh minion, I see:

$ localeLANG=en_GB.UTF-8LANGUAGE=LC_CTYPE="en_US"LC_NUMERIC="en_US"LC_TIME="en_US"LC_COLLATE="en_US"LC_MONETARY="en_US"LC_MESSAGES="en_US"LC_PAPER="en_US"LC_NAME="en_US"LC_ADDRESS="en_US"LC_TELEPHONE="en_US"LC_MEASUREMENT="en_US"LC_IDENTIFICATION="en_US"LC_ALL=en_US

So I get LANG but nothing else. After running salt-call state.sls postgresI see the same thing:

localeLANG=en_GB.UTF-8LANGUAGE=LC_CTYPE="en_US"LC_NUMERIC="en_US"LC_TIME="en_US"LC_COLLATE="en_US"LC_MONETARY="en_US"LC_MESSAGES="en_US"LC_PAPER="en_US"LC_NAME="en_US"LC_ADDRESS="en_US"LC_TELEPHONE="en_US"LC_MEASUREMENT="en_US"LC_IDENTIFICATION="en_US"LC_ALL=en_US

I'm thinking is locale.present enough? Should I set the locale to en_GB.UTF-8 as well?

— Reply to this email directly or view it on GitHubhttps://github.com/saltstack/salt/issues/4543#issuecomment-44459102 .

Øyvind Saltvik

bmcorser commented 10 years ago

Ach, I just bruted it a little.

    ├── locale
    │   ├── env
    │   └── init.sls
    └── postgres.sls
# locale/init.sls
/etc/default/locale:
  file.managed:
    - source: salt://locale/env

en_GB.utf8:
  locale.present
# locale/env
LANG=en_GB.utf8
LC_ALL=en_GB.utf8
# postgres.sls
include:
  - locale

postgres-pkgs:
  pkg.installed:
    - names:
      - postgresql
      - postgresql-server-dev-9.3
      - postgresql-client
    - require:
      - sls: locale
    - reset_system_locale: false

postgresql:
  service.running:
    - require:
      - pkg: postgres-pkgs

postgres-apt:
  pkgrepo.managed:
    - name: "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main"
    - file: /etc/apt/sources.list.d/pgdg.list
    - key_url: http://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc
    - require_in:
        - pkg: postgres-pkgs

Works for me on Umbongo 12.04

rallytime commented 10 years ago

Can this one be closed, or are people still having problems with this?

bmcorser commented 10 years ago

I'm fine for it to be closed.

alexhayes commented 9 years ago

For anyone still experiencing this, like I have, I've raised an issue at https://github.com/saltstack-formulas/postgres-formula/issues/60