aws / elastic-beanstalk-roadmap

AWS Elastic Beanstalk roadmap
https://aws.amazon.com/elasticbeanstalk/
Creative Commons Attribution Share Alike 4.0 International
283 stars 11 forks source link

Python on Amazon Linux 2 platform #15

Closed labisso closed 4 years ago

labisso commented 4 years ago

A Python platform based on Amazon Linux 2 is now available to all customers as a BETA. We'd love it if you could try out the platform and provide any feedback you have. For details about what is different in this platform, check out the documentation page Migrating your Elastic Beanstalk Linux application to Amazon Linux 2.

Feel free to leave comments or complaints in the comments of this issue.

cheng-liven commented 4 years ago

Can you confirm which Python versions are coming to beta AL2 environments on EB? Python 3.6 and/or 3.8?

As PSF accepted the annual release cycle PEP, will you support every new Python version in future? (AWS Lambda supports Python 2.7/3.6/3.7/3.8 currently)

Thanks a lot.

Edit: post the GeoDjango support in a new ticket: https://github.com/aws/elastic-beanstalk-roadmap/issues/28

labisso commented 4 years ago

hi @cheng-liven for AL2 platforms we plan to start with Python 3.7 only, with 3.8 (and future versions) to follow. Do you have a specific need for 3.6?

shyamkgopal commented 4 years ago

Please can share timelines that would be great. I am waiting for quite sometime considering I am already using AL2 across the board but EB is still on AL.

cheng-liven commented 4 years ago

hi @cheng-liven for AL2 platforms we plan to start with Python 3.7 only, with 3.8 (and future versions) to follow. Do you have a specific need for 3.6?

No, I don't have a specific need for Python 3.6, it's just the latest Python 3.x version supported on EB AL 2018.3 platform, so it might be easier for others to migrate to AL2 platforms. It is actually great that you can spend more time assessing #28 other than Python 3.6 support consider how fundamental GIS is in the mobile Internet era.

mattjmorrison commented 4 years ago

Have you thought about switching to use or supporting Pipenv instead of pip & virtualenv by default?

labisso commented 4 years ago

@mattjmorrison Yes! We're planning to use Pipenv in our upcoming Python on Amazon Linux 2 platform.

shyamkgopal commented 4 years ago

AL EOL is 12/31/2020. When do you think we can start using AL2 EB since it will be close-cut between AL EB vs AL2 EB from the consumer perspective.

JustinTArthur commented 4 years ago

Is there a way to access the original aws:elasticbeanstalk:container:python values from within the Procfile, e.g. for determining the process count to use?

JustinTArthur commented 4 years ago

Also, the new docs mention that the hooks have "…access to all environment properties that you've defined in application options". Are there any examples on how to do this from within a platform script? Thanks!

jihoon796 commented 4 years ago

The new docs mention some limitations with the beta:

Amazon Linux 2 platforms are missing some features in the beta versions. We're working on adding support for these features. Here's a list of features that aren't supported at this time.

  • Serving static files
  • Custom platforms

When will these features be available? I am wanting to migrate a Django application.

Sekhar-Kutikuppala commented 4 years ago

Python 3.7 on AL2 platform is now generally available. For more information see What's New post and Release Notes

saranshsingh1 commented 4 years ago

Thank you for the migration Amazon Linux 2 using Python 3.7.

I am wondering if there will be a way to add hooks for config deployment like the ones available on Amazon Linux in future updates since the default config deployment breaks our current deployment since we modify certain configuration settings.

Sekhar-Kutikuppala commented 4 years ago

@saranshsingh1 The Platform hooks feature on AL2 platforms is described in this blog post

saranshsingh1 commented 4 years ago

Yes, but they only work when you deploy the application.

Say, you make some configuration changes like adding environment variables or change the ec2 instance type, etc., that triggers [INFO] Engine command: (config-deploy) which does not use any of the platform hooks.

So, I was wondering if you guys plan to add that feature as well just like how AL1 had appdeploy, configdeploy, and restartappserver. It would help to have configdeploy alongside the current predeploy and postdeploy hooks.

I can share the cloudwatch logs which show the entire deployment process failing because the hooks in predeploy and postdeploy hooks are not executed, rightfully so.

Sekhar-Kutikuppala commented 4 years ago

@saranshsingh1 Thanks for the feedback. The configdeploy is currently not available in AL2 platforms and we will evaluate this request.

saranshsingh1 commented 4 years ago

Thank you. I appreciate the response and hopefully this feature does get added.

N3Cr0 commented 4 years ago

Hi, just wondering if anyone else is having problems with deployment on EB AL2?

Was fighting with it all day yesterday and cannot get apps to deploy properly. Here's what i've tried:

If i switch back to the sample it works again. I just cant figure this out. web.service failed to start.

Help! :)

saranshsingh1 commented 4 years ago

@N3Cr0 Look at the logs of eb-engine.log. That should give you an idea of what went wrong.

N3Cr0 commented 4 years ago

@N3Cr0 Look at the logs of eb-engine.log. That should give you an idea of what went wrong.

Thanks @saranshsingh1 . It's pretty non-descript on this issue.

But that said, I think I figured out what I was doing wrong (maybe you can confirm).

I didn't realise that so much changed between AL1 and AL2 for Python. And following tutorials was actually to my detriment. I was still using Pip and 'requirements.txt' where I should have been using Pipfile. Maybe some extras config required too.

The AWS sample App I downloaded originally was the one foe AL1. So it didn't work either. Am I correct?

saranshsingh1 commented 4 years ago

The AWS sample App I downloaded originally was the one foe AL1. So it didn't work either. Am I correct?

@N3Cr0 Sorry, I never downloaded the sample application, so I cannot confirm that.

I was still using Pip and 'requirements.txt' where I should have been using Pipfile. Maybe some extras config required too.

You can use either requirements.txt or Pipfile to specify dependencies. Note the order mentioned in the last paragraph. Either use requirements.txt or Pipfile (based on your preference).

There are various other changes like wsgi and static file naming convention, using a Procfile to specify the gunicorn command, etc.

I will say that I found the migration guide a little useful.

digicase commented 4 years ago

I cannot get my Django app to deploy on the new Python platform. Running the command python manage.py collectstatic --noinput fails, both in the old config file style and in the new .platform/hooks/predeploy/collectstatic.sh setup.

Either way Django is not being found as a Python package, as if the virtual environment has not beed activated. Do we now have to activate the virtual environment first?

digicase commented 4 years ago

Here's my shell script

#!/bin/bash
sudo source staging-LQM1lest/bin/activate
python manage.py collectstatic --noinput

And the output from the logs

2020/04/26 15:43:40.142452 [INFO] Running command .platform/hooks/predeploy/collect_static.sh
2020/04/26 15:43:40.162781 [ERROR] sudo: source: command not found
  File "manage.py", line 25
    ) from exc
         ^
SyntaxError: invalid syntax

I have checked on the instance, and Django was successfully install from my requirements.txt file, with django-admin in the bin folder and all the requirements in the site-packages folder for the virtual environment.

Can anyone help me to work out how to get this app deployed? Thanks.

EDIT: sudo source was wrong, removing sudo worked to activate the env.

digicase commented 4 years ago

I checked within the /var/app/staging folder to see what was going on. If I set up a Python prompt I get this:

[ec2-user@ip-xxx-xx-xx-xxx ~]$ cd /var/app/
[ec2-user@ip-xxx-xx-xx-xx app]$ ls
current  staging  venv
[ec2-user@ip-xxx-xx-xx-xx app]$ cd staging/
[ec2-user@ip-xxx-xx-xx-xx staging]$ python
Python 2.7.16 (default, Dec 12 2019, 23:58:22)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>

That shows the virtualenv is not being used, so no wonder Django could not be found. Is the build system on these images missing steps to make sure the venv is being used at all times?

saranshsingh1 commented 4 years ago

@digicase The way beanstalk sets up the virtual environment using pipenv does not help in accessing the virtual environment so we you cannot launch it using pipenv shell.

I cannot get my Django app to deploy on the new Python platform. Running the command python manage.py collectstatic --noinput fails, both in the old config file style and in the new .platform/hooks/predeploy/collectstatic.sh setup

To collect the static files, you have to be in the app directory. If you're running the shell script from predeploy hooks directory, it is the staging directory or current directory for postdeploy hooks.

Try something along the lines of:

  #!/bin/bash

  source /var/app/venv/*/bin/activate
  cd /var/app/staging
  python manage.py collectstatic --noinput

You do not need to add sudo since all the hooks are run as root.

For deployment:

The workaround I am using is getting the PATH environment variable from /opt/elasticbeanstalk/deployment/env, storing it in /var/app/current/path_file and using it as EnvironmentFile=/var/app/current/path_file in the systemd service block.

I have replaced the default web.service file (which beanstalk creates) as below:

  [Unit]
  Description=Gunicorn daemon for application server
  After=network.target

  [Service]
  # systemd.exec ? Execution environment configuration
  User=webapp
  Group=nginx
  Type=notify

  WorkingDirectory=/var/app/current/

  ExecStart=/bin/sh -c "gunicorn -c gunicorn_config.py app_name.wsgi:application"
  ExecStartPost=/bin/sh -c "systemctl show -p MainPID web.service | cut -d= -f2 > /var/pids/web.pid"
  ExecStopPost=/bin/sh -c "rm -f /var/pids/web.pid"
  Restart=always
  EnvironmentFile=/var/app/current/path_file

  # https://www.freedesktop.org/software/systemd/man/systemd.exec.html#SyslogIdentifier=
  StandardOutput=syslog
  StandardError=syslog
  SyslogIdentifier=web
  KillMode=mixed

  [Install]
  WantedBy=multi-user.target

Feel free to use the above as a reference, works perfectly for me. The above setup of getting the PATH from the env file ensures I am not hardcoding the virtual environment path.

digicase commented 4 years ago

Thanks @saranshsingh1 I'll try this.

james-yun commented 4 years ago

Is there any way to get the leader_only functionality from container commands using platform hooks?

cristianoccazinsp commented 4 years ago

Is there any way to get the leader_only functionality from container commands using platform hooks?

Found a way to do that?

james-yun commented 4 years ago

Is there any way to get the leader_only functionality from container commands using platform hooks?

Found a way to do that?

Nope, would have been useful for django migration. Although inefficient, my workaround is to just migrate on every ec2 deployment. This works for me. .platform/hooks/predeploy/01_django.sh

#!/bin/bash
source /var/app/venv/*/bin/activate
python manage.py migrate
python manage.py collectstatic --noinput
cristianoccazinsp commented 4 years ago

Interesting. I assume this only works because ELB deploys one instance at a time?

james-yun commented 4 years ago

The predeploy hook runs on every new instance that ELB deploys. Doesn't necessarily have to be one at a time (depends on if you're doing rolling deployments or all at once). Should work in either case.

cristianoccazinsp commented 4 years ago

@saranshsingh1 what would be the nginx configuration files look like in order to add the static files location, and various extra settings:

Trying to achieve the following settings, but it seems rather impossible without fully re-writing the nginx config file:

saranshsingh1 commented 4 years ago

@cristianoccazinsp As the documentation for Reverse configuration proxy suggests, you can add/extend the Nginx configuration as you wish.

For application related settings like:

static files path (django) add cache headers to static files http to https redirect if forwarded-for proto pass forwarded-for-proto header to the proxy Add security headers (X-Frame-Options, HSTS, etc..) to both static and proxied requests setup GZIP to both static and proxied requests

you can add the properties in a *.conf file in the .platform/hooks/nginx/conf.d directory. I have different files for gzip, proxy headers (like x-forwarded-for, etc), security headers (like x-frame-options, etc), and application (like static file path, redirects, etc). All these files are placed in the conf.d directory.

You can have a similar approach to disable server_tokens or other properties of http block by adding these properties using include keyword of nginx but I found it easier to just replace the complete nginx.conf file.

cristianoccazinsp commented 4 years ago

It's unclear from the docs if these extra settings are added below the server { ... } group or under the location / { ... } group.

How would you add settings that are supposed to be below server and others below location?

digicase commented 4 years ago

@cristianoccazinsp Here's the /etc/nginx/nginx.conf file I found.

#Elastic Beanstalk Nginx Configuration File

user                    nginx;
error_log               /var/log/nginx/error.log warn;
pid                     /var/run/nginx.pid;
worker_processes        auto;
worker_rlimit_nofile    66485;

events {
    worker_connections  1024;
}

http {
    include   /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    include   conf.d/*.conf;

    map $http_upgrade $connection_upgrade {
        default     "upgrade";
    }

    server {
    listen        80 default_server;
        access_log    /var/log/nginx/access.log main;

        client_header_timeout 60;
        client_body_timeout   60;
        keepalive_timeout     60;
        gzip                  off;
        gzip_comp_level       4;
        gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml appl$

        # Include the Elastic Beanstalk generated locations
        include conf.d/elasticbeanstalk/*.conf;
    }
}
digicase commented 4 years ago

@labisso it would be really helpful to get the AWS EB docs updated to include more detail on setting up a Python app in EB on the AL2 platform. The things that have been brought up in the comments in this thread, especially leader_only functionality for platform hooks. Activating the Python virtual environment and using environment vars in platform hooks. Documenting the default Nginx config. Thanks.

serebrov commented 4 years ago

Regarding the leader_only in scripts, it is possible to do something like this in the EB config:

container_commands:
  001-command:
    command: echo "ELASTICBEANSTALK_CMD_LEADER=true" > .ebextensions/.elastic-beanstalk-cmd-leader
    leader_only:    true

And then just check for if that file exists or source it.

On the old platform, I've also used this snippet:

    if [[ "$EB_IS_COMMAND_LEADER" == "true" ]]; then
        # leader
    else
        # not a leader
    fi

The code was in the /opt/elasticbeanstalk/bin/leader-test.sh and the EB_IS_COMMAND_LEADER variable was set during deployment. I can still see this script on the new platform, in the /opt/elasticbeanstalk/config/private/containercommandsupport/leader-test.sh, so this method might also work on the new platform, I didn't have time to check it yet.

cristianoccazinsp commented 4 years ago

Thanks for the nginx file, but that still doesn't really tell me how can I add various settings under various places of the config file.

For example, the default configuration for the python reverse proxy only adds X-Forwarded-For but not X-Forwarded-For-Proto, so I need to add an extra line under that location / {...} section.

digicase commented 4 years ago

I’ll have to leave it to those with Nginx experience to advise on that. I hope Amazon can update their docs with more info and specific examples. For the previous Linux platforms there is a Github repository of proxy server config examples.

saranshsingh1 commented 4 years ago

The Nginx configuration really depends on how you want the different configurations to work with each other.

I have my main config file which includes all other options. Below is my main file.

/etc/nginx/nginx.conf

.....

http {
    charset  utf-8;
    sendfile on;
    tcp_nopush  on;
    tcp_nodelay on;
    server_tokens off;
    log_not_found off;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    client_max_body_size 100M;

    # logging
    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

    # load configs
    include   conf.d/*.conf;    # this is the reference location

    # MIME
    include   /etc/nginx/mime.types;

    ....
}

Then all my server blocks for different URLs are placed inside the conf.d directory like conf.d/default.conf, conf.d/app1.conf, conf.d/app2.conf, etc. Below is one such file for /etc/nginx/conf.d/app1.conf:

# http -> https server block
server {
    listen                     80;
    listen                     [::]:80;

    server_name                app_url.com;

    return                     301 https://app_url.com$request_uri;
}

# Subdomains redirect
server {
    listen                     443 ssl http2;
    listen                     [::]:443 ssl http2;

    server_name                *.app_url.com;

    # Include the SSL certificates
    include                    conf.d/common/ssl_cert.conf;

    return                     301 https://app_url.com$request_uri;
}

# main web config
server {
    listen                     443 ssl http2;
    listen                     [::]:443 ssl http2;

    server_name                app_url.com;

    # Include the Elastic Beanstalk generated locations
    include                    conf.d/common/*.conf;
}

With the above setup, you can add whatever properties you want. I use the above setup following the DRY principle.

Adding the rest of the files for completeness the answer.

/etc/nginx/conf.d/common/application.conf

# Set base variable to application directory for reference
set $base                /var/app/current;

# Django static
location /static/ {
    alias                $base/static/;
}

# Django media
location /media/ {
    alias                $base/media/;
}

# Django reverse proxy
# These values are passed to app server
location / {
    proxy_http_version   1.1;
    proxy_cache_bypass   $http_upgrade;

    proxy_set_header     Upgrade               $http_upgrade;
    proxy_set_header     Connection            "upgrade";
    proxy_set_header     Host                  $host;
    proxy_set_header     X-Real-IP             $remote_addr;
    proxy_set_header     X-Forwarded-For       $proxy_add_x_forwarded_for;
    proxy_set_header     X-Forwarded-Proto     $scheme;
    proxy_set_header     X-Forwarded-Host      $host;
    proxy_set_header     X-Forwarded-Port      $server_port;

    proxy_set_header     X-Request-ID          $request_id; # Pass to app server  --- Maybe add this
    proxy_redirect       off;

    proxy_pass           http://unix:/run/my_app/gunicorn.sock;
}

# additional config
 favicon.ico
 location = /favicon.ico {
     log_not_found       off;
     access_log          off;
 }

# robots.txt
location = /robots.txt {
    log_not_found       off;
    access_log          off;
}

/etc/nginx/conf.d/common/headers.conf

# Security Headers
add_header      X-Request-ID                    $request_id;
add_header      X-Frame-Options                 "SAMEORIGIN" always;
add_header      X-XSS-Protection                "1; mode=block" always;
add_header      Referrer-Policy                 "no-referrer-when-downgrade" always;
add_header      Strict-Transport-Security       "max-age=31536000; includeSubDomains; " always;
# add_header      Content-Security-Policy         "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
# add_header      X-Content-Type-Options          "nosniff" always;     # nginx already adds; avoid duplication

/etc/nginx/conf.d/common/gzip.conf

# gzip
gzip                on;
gzip_vary           on;
gzip_proxied        any;
gzip_comp_level     6;
gzip_types          text/plain text/css text/xml application/json application/javascript application/rss+xml application/atom+xml image/svg+xml;
cristianoccazinsp commented 4 years ago

@saranshsingh1 That's awesome, thanks!

BrianPainter commented 4 years ago

attempting to run .platform/hooks/predeploy/01_django.sh and getting permission denied in the logs. file is below:

#!/bin/bash source /var/app/venv/*/bin/activate cd /var/app/staging/ python manage.py migrate python manage.py collectstatic --noinput

I can run the commands manually as root when I ssh into the instance, but not with the predeploy hook.

cristianoccazinsp commented 4 years ago

Silly that ELB doesn't do this automatically, but you can fix that by adding the following to your .ebextensions:

container_commands:

  001-chmd:
    command: chmod -R 777 .platform/
empty commented 4 years ago

Silly that ELB doesn't do this automatically, but you can fix that by adding the following to your .ebextensions:

container_commands:

  001-chmd:
    command: chmod -R 777 .platform/

That's the wrong move. The documentation clearly says you have to add execute rights to the shell script via:

chmod +x .platform/hooks/predeploy/01_django.sh

empty commented 4 years ago

Here's something that might be helpful for some. I ran into an issue where Django wouldn't see any of the environment variables that were set on the ElasticBeanstalk configuration when running management commands or cronjobs. With Amazon Linux you could source /opt/python/current/env and that would export all the environment variables. On Amazon Linux 2 that file doesn't exist.

I created a post deploy script in .platform/hooks/postdeploy/01_env.sh that recreates the old file on deploy.

Put these contents in the shell script:

#!/bin/bash

input="/opt/elasticbeanstalk/deployment/env"
output="/var/app/env"

> $output

while IFS= read -r line
do
  key=`echo $line | cut -d= -f1`
  value=`echo $line | cut -d= -f2`
  echo "export $key=\"$value\"" >> $output
done < "$input"

chmod 444 $output

You can use tighter security if you want, but just remember that root will create this file and when you ssh into the server you are the ec2-user.

Don't forget to chmod +x .platform/hooks/postdeploy/01_env.sh or the build system will not have rights to execute the script.

thibulko commented 4 years ago

Here's something that might be helpful for some. I ran into an issue where Django wouldn't see any of the environment variables that were set on the ElasticBeanstalk configuration when running management commands or cronjobs. With Amazon Linux you could source /opt/python/current/env and that would export all the environment variables. On Amazon Linux 2 that file doesn't exist.

I created a post deploy script in .platform/hooks/postdeploy/01_env.sh that recreates the old file on deploy.

Put these contents in the shell script:

#!/bin/bash

input="/opt/elasticbeanstalk/deployment/env"
output="/var/app/env"

> $output

while IFS= read -r line
do
  key=`echo $line | cut -d= -f1`
  value=`echo $line | cut -d= -f2`
  echo "export $key=\"$value\"" >> $output
done < "$input"

chmod 444 $output

You can use tighter security if you want, but just remember that root will create this file and when you ssh into the server you are the ec2-user.

Don't forget to chmod +x .platform/hooks/postdeploy/01_env.sh or the build system will not have rights to execute the script.

It don't work for me, because script .platform/hooks/postdeploy/01_env.sh run before save variables to /opt/elasticbeanstalk/deployment/env.

I observe this when I add a new environment properties through the Beanstalk web console.

packerbacker3333 commented 4 years ago

Has anyone experienced permission denied on the .sh file even after running chmod +x ? I ran the script locally to confirm it could be executed. I also ssh'd into the EC2 instance and ran all of the lines manually so I know they work. But when I deploy the app, it still fails and says that permission is denied for the .sh file.

nick-brady commented 4 years ago

@labisso it would be really helpful to get the AWS EB docs updated to include more detail on setting up a Python app in EB on the AL2 platform. The things that have been brought up in the comments in this thread, especially leader_only functionality for platform hooks. Activating the Python virtual environment and using environment vars in platform hooks. Documenting the default Nginx config. Thanks.

Is there a timeframe for this? The comment was made months ago and even little tidbit improvements would be massively helpful in avoiding wasted time. I was unfamiliar with AL1 & AL2 and have gone through a week of pain due to outdated documentation and major differences I didn't know I had to look out for as I'm fleshing out an eb deployment hitting snag after snag. Even just documenting which log files contain what would be so helpful - I finally realized after opening random log files on the machine that /var/log/cfn-init-cmd.log is the one I wanted for my last issue.

I found out from this log my command failing is due to the python environment not loading how it used to. How do I execute a command that uses the correct python environment in AL2? Is using a command no longer the recommended way to do a migrate in Django? It would be very helpful to know how the python environment is intended to be used in AL2 - especially for such a widely used framework such as Django.

Alex3917 commented 4 years ago

Since no one has mentioned it yet here, the docs for installing CloudWatch on Beanstalk are super outdated and should get updated:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html

Also I created gists for how I get Celery and nginx working:

https://gist.github.com/Alex3917/9b09e67069c88361d4e0187e1968bc64 https://gist.github.com/Alex3917/de07c7e181b61133662af5c76d3ea869

appli-intramuros commented 4 years ago

Hello,

I try to execute a run_collectstatic.sh file in .platform/confighooks/predeploy during configuration deployment (not application deployment, Configuration deployment hooks)

My files: .platform/confighooks/predeploy/run_collectstatics.sh:

#!/usr/bin/env bash
source /var/app/venv/*/bin/activate
python manage.py collectstatic --noinput

.ebextensions/01_permission.config:

container_commands:  
    01_change_mode_of_hooks_file:
        command: chmod +x .platform/hooks/predeploy/run_collectstatics.sh
    02_change_mode_of_confighooks_file:
        command: chmod +x .platform/confighooks/predeploy/run_collectstatics.sh

But the configuration deployment fail with the following error (the application deployment is OK): Error: Command .platform/confighooks/predeploy/run_collectstatics.sh failed with error fork/exec .platform/confighooks/predeploy/run_collectstatics.sh: permission denied

It seems that container_commands is not executed during configuration deployment. How can I change the chmod of the file for configuration deployment ? Any alternative ?

olifrieda commented 4 years ago

@appli-intramuros : As the .platform folder is part of your artifact, you should add the executable flag before you upload or pack your zip-file. So run the chmod command locally before you create a new version into your EBS environment.