CRM token and baseUrl config issue

[A-linea]

Good afternoon!

My workflow crm-skeleton mailer-skeleton remp sso which generates a token for Mailer Everything is set up according to the documentation As an example.

mailer-skeleton config.neon

php:
    intl.default_locale: en_EN

parameters:
    crm:
        addr: @environmentConfig::get('CRM_ADDR')
        api_token: @environmentConfig::get('CRM_API_TOKEN')
services:
    - Remp\MailerModule\Models\Users\Crm(%crm.addr%, %crm.api_token%)
    authenticator:
        factory: Remp\MailerModule\Models\Auth\SimpleAuthenticator
        setup:
            - addUser('admin@admin.sk', 'passphrase_change_me')    

    # This is a reference IUser implementation.
    # You should make your own implementation against your CRM system and register it here.

    # To be able to send an email, you need to provide at least one ISegment implementation
    # You should make your own implementation against your CRM system and register it here.
    segmentAgreggator:
        setup:
#            - register(Remp\MailerModule\Models\Segment\Dummy())
            - register(Remp\MailerModule\Models\Segment\Crm(%crm.addr%, %crm.api_token%))
#           - register(Remp\MailerModule\Models\Segment\Beam(%remp.beam.segments_addr%))
#           - register(Remp\MailerModule\Models\Segment\Pythia(%remp.pythia.segments_addr%))

    # All console commands you need to use should be registered here
    # This is a set that we recommend to use
    console:
        setup:
            - add(Remp\MailerModule\Commands\SyncUserSubscriptionsCommand())
            - add(Remp\MailerModule\Commands\MailgunEventsCommand())
            # Sample command - servers as an example of adding custom code to Mailer
            - add(Remp\Mailer\Commands\SampleCommand())
            # If you're using Beam, you can enable this command to pull conversions from there
            # - add(Remp\MailerModule\Commands\ProcessConversionStatsCommand())

    embedParser:
        setup:
            - setVideoLinkText("Click to play video")

    articleLocker:
        setup:
            - setLockText("Exclusive content")
            - setupLockLink("Registration", "https://www.demosite.com/registration")

    unsubscribeDroppedHandler:
        factory: Remp\MailerModule\Hermes\UnsubscribeDroppedHandler
        setup:
            - setThreshold(3) # number of dropped emails before unsubscribing user

    # shutdown hermes and mail worker with file (checks modification date; see SharedFileShutdown implementation)
    hermesShutdown: Tomaj\Hermes\Shutdown\SharedFileShutdown('/tmp/hermes_shutdown')

    hermesWorker:
        setup:
            - add('email-dropped', @unsubscribeDroppedHandler)

            # Enable these only if Remp\MailerModule\Tracker\Remp (or other Itracker implementation) is enabled as well.
#            - add('mail-sent', Remp\MailerModule\Hermes\MailSentHandler())
#            - add('user-subscribed', Remp\MailerModule\Hermes\TrackSubscribeUnsubscribeHandler())
#            - add('user-unsubscribed', Remp\MailerModule\Hermes\TrackSubscribeUnsubscribeHandler())

            # Notifications to REMP CRM that users' newsletter subscription was updated.
#            - add('user-subscribed', Remp\MailerModule\Hermes\NotifyCrmSubscribeUnsubscribeHandler())
#            - add('user-unsubscribed', Remp\MailerModule\Hermes\NotifyCrmSubscribeUnsubscribeHandler())
#            - add('user-subscribed-variant', Remp\MailerModule\Hermes\NotifyCrmSubscribeUnsubscribeHandler())
#            - add('user-unsubscribed-variant', Remp\MailerModule\Hermes\NotifyCrmSubscribeUnsubscribeHandler())

    # Setup loggers output
    commandsLogger:
        setup:
            - pushHandler(Monolog\Handler\StreamHandler("%appDir%/../log/commands.log"))
    mailProcessLogger:
        setup:
            - pushHandler(Monolog\Handler\StreamHandler("%appDir%/../log/mail_process_job.log"))
    hermesLogger:
        setup:
            - pushHandler(Monolog\Handler\StreamHandler("%appDir%/../log/hermes.log"))
            - pushHandler(Monolog\Handler\ErrorLogHandler())

    crmClient:
           factory: Remp\MailerModule\Models\Crm\Client(%crm.addr%, %crm.api_token%)         

#    - Remp\MailerModule\Tracker\Remp(%remp.beam.tracker_addr%, %remp.beam.tracker_property_token%)

#    eventDispatcher:
#        factory: League\Event\EventDispatcher
#        setup:
#            - subscribeTo(Remp\MailerModule\Events\MailSentEvent, Remp\MailerModule\Events\MailSentEventHandler())

    # If you're using Beam and want to use ProcessConversionStatsCommand, enable this repository (it's a dependency)
#    - Remp\MailerModule\Repositories\BeamConversionsRepository

    # Setup of multiple mailers of the same type
#   mailFactory:
#       setup:
#           - addMailer(Remp\MailerModule\Models\Mailer\MailgunMailer(), 'us')
#           - addMailer(Remp\MailerModule\Models\Mailer\MailgunMailer(), 'eu')

# local_configs override DBs configs records.
local_configs:
    # Configuring usage of mailhog for local development
    default_mailer: remp_smtp
    remp_smtp_host: 127.0.0.1
    remp_smtp_port: 1025
    remp_smtp_username: ''
    remp_smtp_password: ''
    remp_smtp_secure: ''

# If running Mailer on Valet, please uncomment the following lines and adjust the full path to REMP/Mailer folder
#webpack:
#    build:
#        directory: /FULL_PATH_TO_REMP_FOLDER/Mailer/www/assets/vendor

---

mailer-skeleton .env

# Environment name. If set to "local", all debug options are enabled by framework automatically.
ENV=local

# Application locale affecting formatting settings of viewed data.
LOCALE=en_US

# Default timezone.
TIMEZONE="Europe/Kiev"

# Flag whether the framework should enforce HTTPS. It will set additional $_SERVER variables.
# See app/bootstrap.php for specific usage.
FORCE_HTTPS=false

# URL to frontend page providing unsubscr`ibe feature for given %type% of newsletter. Usually used to direct user from
# email's "unsusbcribe" button. When entered, {{ unsubscribe }} variable can be used within all email templates instead
# of manually entering unsubscribe URL everywhere.
UNSUBSCRIBE_URL=https://mailer.press/mail/mail-settings/un-subscribe-email/%type%

# URL to frontend page providing all email related settings - usually page with possibility to subscribe and unsubscribe
# to different newsletters. When entered, {{ settings }} variable can be used within all email templates instead of
# manually entering settings URL everywhere.
SETTINGS_URL=https://mailer.press/email-settings

#####################
## SQL database connection details. MySQL (and derivates) are supported and tested.

# Database adapter implementation to be used.
# - default set to "mysql", keep without change unless you want to use your own implementation
DB_ADAPTER=mysql

# Database host (e.g. localhost, IP address or any resolvable host of your database server instance).
DB_HOST=localhost

# Name of the database.
DB_NAME=mailer

# User to be used to connect to database.
DB_USER=root

# Password to be used to connect to database.
DB_PASS=3661269

# Port to be used to connect to database. 3306 is the default port used by MySQL and its derivates.
DB_PORT=3306

#####################
## Redis connection details

# Redis host (e.g. localhost, IP address or any resolvable host of your redis server instance).
REDIS_HOST=localhost

# Redis connection port. 6379 is the default port used by Redis installation.
REDIS_PORT=6379

# Default redis DB
REDIS_DB=1

# Redis connection password. Enable and configure if your Redis instance requires authentication.
REDIS_PASSWORD=123456

#####################
## SSO configuration

# Full address of SSO instance
SSO_ADDR=http://sso.remp.press

# URL where user is redirected if SSO login fails
SSO_ERROR_URL=http://mailer.press/sign/error

#####################
## REMP services
## Following are addresses of other installed REMP services. Keep blank to indicate that the service is not used.

# REMP_BEAM_ADDR=http://beam.remp.press
# REMP_CAMPAIGN_ADDR=http://campaign.remp.press
# REMP_BEAM_SEGMENTS_ADDR=http://beam_segments:8082/
# REMP_BEAM_TRACKER_ADDR=http://beam_tracker:8081/
# REMP_PYTHIA_SEGMENTS_ADDR=http://pythia_segments:8083/

# Property token from Beam (go to Beam - Properties to get the token) used for event tracking
#REMP_BEAM_TRACKER_PROPERTY_TOKEN=

# API token to access Beam. The default configuration of REMP tools always uses API token generated in SSO, you can
# grab your token there.
#REMP_BEAM_API_TOKEN=

#####################
## Application options

#  Maximum number of returned search results
#
#  This value represents limit for number of returned search results.
#  IMPORTANT: this number affects each searchable entity separately
#  e.g.: when MAX_RESULT_COUNT is being set to 5 and you search
#  model_1 and model_2 you can get max 10 results
#SEARCH_MAX_RESULT_COUNT=5

# Default mail template editor.
#
# Available values are `wysiwyg` or `codemirror`.
TEMPLATE_EDITOR=codemirror
CRM_ADDR=http://crm.press
CRM_API_TOKEN=crm_press_api_token

---

crm-skeleton config.neon

# SECURITY WARNING: it is CRITICAL that this file & directory are NOT accessible directly via a web browser!
#
# If you don't protect this directory from direct web access, anybody will be able to see your passwords.
# http://nette.org/security-warning
#
parameters:
    errbit:
        send_errors: false

    redis:
        host: @environmentConfig::get('CRM_REDIS_HOST')
        port: @environmentConfig::get('CRM_REDIS_PORT')
        password: @environmentConfig::get('CRM_REDIS_PASSWORD')
        database: @environmentConfig::get('CRM_REDIS_DATABASE')
extensions:
    local_configs: Crm\ApplicationModule\Config\ConfigExtension
    translation: Contributte\Translation\DI\TranslationExtension
    - Crm\ApplicationModule\DI\ApplicationModuleExtension
    - Crm\ApiModule\DI\ApiModuleExtension
    - Crm\AdminModule\DI\AdminModuleExtension
    - Crm\DashboardModule\DI\DashboardModuleExtension
    users: Crm\UsersModule\DI\UsersModuleExtension
    - Crm\SubscriptionsModule\DI\SubscriptionsModuleExtension
    - Crm\SegmentModule\DI\SegmentModuleExtension
    - Crm\PaymentsModule\DI\PaymentsModuleExtension
    - Crm\SalesFunnelModule\DI\SalesFunnelModuleExtension
    - Crm\InvoicesModule\DI\InvoicesModuleExtension
    - Crm\IssuesModule\DI\IssuesModuleExtension
    - Crm\PrintModule\DI\PrintModuleExtension
    - Crm\ProductsModule\DI\ProductsModuleExtension
    - Crm\UpgradesModule\DI\UpgradesModuleExtension
    - Crm\ScenariosModule\DI\ScenariosModuleExtension
    - Crm\OnboardingModule\DI\OnboardingModuleExtension
    - Crm\GiftsModule\DI\GiftsModuleExtension
    remp_mailer: Crm\RempMailerModule\DI\RempMailerModuleExtension
#   hermes: Crm\HermesModule\DI\HermesModuleExtension
    #- Crm\StripeModule\DI\StripeModuleExtension

# parameters for extensions
users:
    countries:
        default: @environmentConfig::get('CRM_DEFAULT_COUNTRY_ISO')

translation:
    dirs:
        - %appDir%/lang
    locales:
        default: en_US
        whitelist: [en_US, sk_SK, cs_CZ]
    localeResolvers:
        - Contributte\Translation\LocalesResolvers\Parameter
        # Router translation resolver needs to be disabled by default
        # in order to migrate the project's database. Router resolver would
        # try to initialize routes of all of the modules possibly attempting
        # to read configuration from the database (like products module does).
        # That would fail because the DB would be empty.
        #
        # Feel free to enable this back after the initial run of the command
        # php bin/command.php phinx:migrate

        #- Contributte\Translation\LocalesResolvers\Router
    loaders:
        neon: Contributte\Translation\Loaders\Neon
        yml: Symfony\Component\Translation\Loader\YamlFileLoader

php:
    date.timezone: Europe/Kyiv
    intl.error_level: 2 # E_WARNING

security:
    debugger: true

application:
    errorPresenter: Application:Error
    mapping:
        *: Crm\*Module\Presenters\*Presenter
    # TODO: we need to scan composer to properly register extensions
    scanComposer: yes

session:
    expiration: 14 days
    autoStart: smart

http:
    frames: yes

tracy:
    netteMailer: no

database:
    default:
        dsn: @environmentConfig::getDsn()
        user: @environmentConfig::get('CRM_DB_USER')
        password: @environmentConfig::get('CRM_DB_PASS')
        options:
            lazy: yes
services: 
    moduleManager:
            setup:
                - addModule(Crm\FondyModule\FondyModule())

    - Crm\FondyModule\Api\FondyApiHandler
    - Crm\FondyModule\Seeders\ConfigSeeder
    - Crm\FondyModule\Seeders\PaymentGatewaysSeeder
    - Crm\FondyModule\Seeders\SalesFunnelsSeeder    
    - Crm\FondyModule\Gateways\Fondy
    - Crm\FondyModule\Gateways\FondyRecurrent
    - Crm\FondyModule\Models\Repository\FondyPaymentsRepository
    - Crm\FondyModule\Components\PaymentError\PaymentError

    gatewayFactory:
            setup:
                - registerGateway(fondy, Crm\FondyModule\Gateways\Fondy)
                - registerGateway(fondy_recurrent, Crm\FondyModule\Gateways\FondyRecurrent)

remp_mailer:
    # Base URL where mailer is hosted.
    # host: http://liga-crm-mailer.liga.net
    host: http://mailer.press
    # API token to communicate with Mailer. By default the token can be acquired in REMP SSO.
    api_token: mailer_token_BY_SSO

I've encountered a strange behaviour, when I create a new scenario in crm-skeleton it pulls all the segments and everything seems to be ok, but after I save and edit this scenario, it is infinite loading. As you can see on the screenshot. With 504 errors. In the log, we see the following

`
[2023-10-18 12-49-30] GuzzleHttp\Exception\ServerException: Server error: `GET http://mailer.press/api/v1/mailers/templates?mail_type_codes%5B0%5D=system&mail_type_codes%5B1%5D=system_optional&with_mail_types=1` resulted in a `500 Internal Server Error` response: <!DOCTYPE html><!-- "' --></textarea></script></style></pre></xmp></a></iframe></noembed></noframes></noscript></option> (truncated...) #500 in /Users/eugene/Work/crm-skeleton/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113  @  http://crm.press//api/v1/mail-template/list  @@  error--2023-10-18--09-29--414a308013.html`

I triggered this endpoint by Postman If I pass the api_key for the crm_skeleton, it works and I see expected response, but when I pass api_key for the mailer-skeleton, it does not.

On the other hand, if I comment out the line in

• Remp\MailerModule\Models\Users\Crm(%crm.addr%, %crm.api_token%) or comment out

  CRM_ADDR

  CRM_API_TOKEN

  then everything is loaded on the scripts page in crm-skeleton, but then the mailer itself does not work accordingly because it needs this data.

Please, tell me what is wrong with my settings?

And one more question: when I request mailer-template-list from Postman, I get an empty array, where should I look? As I can see from the database, they related also to mail_type_category, maybe i need additional param for category. I mean, when I want to select crm-skeleton -> actions->sendEmail in the scenario, there is nothing. How to bind mail templates, from mailer to crm.

Thank you!

[A-linea]

Small update I tried to dig deeper as much as I could, wrote down the data in an html file what was happening in Guzzle inside /vendor/Guzzle and this is the response that crm receives from mailer, and that's why everything is falling down

Well, it was due to indentation, after the fix I get the following response that the mailer receives from sso

UPD:

Hi Peter, in order to make sure where exactly the problem is occurring, I followed the whole chain to /var/www/remp/Sso/app/Http/Controllers/AuthController.php and found out that it receives the token and sends a response, but it takes a very long time. The problem itself occurs in /var/www/remp/Mailer/vendor/remp/mailer-module/src/Models/Sso/Client.phpon line 73. This is where the token gets instantly. Something wrong since the request from Guzzle. I tested in three variants.

1. In the docker + crm-skeleton
2. remp repository + crm-skeleton without a docker, installed and configured manually.
3. Separately mailer-skeleton, crm-skeleton sso. Also for exapmle if I return simple return true from /var/www/remp/Mailer/vendor/remp/mailer-module/src/Models/Sso/Client.php validToken():boolthis method sends a response instantly.

[rootpd]

1) This is very weird, SSO is basically an empty Laravel application with a very basic feature-set. The gateway timeout could possibly occur in Docker if you put your computer to sleep and then woke up. Something with the Docker's network isn't right (or maybe we have something wrongly configured) and the containers don't see each other correctly.

2) Since you use CRM, you did right by uncommenting CRM's IUser interface implementation. The idea is that Mailer needs primary source of users. By default, it's provided by Dummy implementation which only contains two users. Linking it to CRM is a good idea.

For now, make sure that crm_press_api_token that you entered in Mailer's .env file has full set of API rights. Each API key needs to have the API endpoints it can access enabled manually. You can find it in the CRM - API - Api access menu.

3) Empty mail set is OK, unless you already created some emails in Mailer or seeded them with make install-demo. Also, the CRM doesn't request every email in Mailer, but (by default) only emails in the system or system_optional mail categories. You don't want your weekly newsletters to appear in scenarios, you only want your transactional (system) emails there.

4) Regarding slow SSO response - you can try to bypass that for now by configuring Mailer not to use SSO to verify that API key is valid, but to resolve that locally. Add this to your config.neon or config.local.neon:

services:
        apiTokenRepository:
                factory: Tomaj\NetteApi\Misc\StaticTokenRepository(['your_custom_secret': '*'])

If there is an API request with Authorization: Bearer your_custom_secret, Mailer will accept that as valid.

[A-linea]

Thanks for your response.

1. I have already tested this in different versions, please see above: I followed the whole chain to /var/www/remp/Sso/app/Http/Controllers/AuthController.php and found out that it receives the token and sends a response, but it takes a very long time. The problem itself occurs in /var/www/remp/Mailer/vendor/remp/mailer-module/src/Models/Sso/Client.php on line 73. This is where the token gets instantly. Something wrong since the request from Guzzle. I tested in three variants.

In the docker + crm-skeleton remp repository + crm-skeleton without a docker, installed and configured manually. Separately mailer-skeleton, crm-skeleton sso. Also for exapmle if I return simple return true from /var/www/remp/Mailer/vendor/remp/mailer-module/src/Models/Sso/Client.php validToken():bool this method sends a response instantly. Also the check-token endpoint works instantly if I send a request from Postman

2. So all the tokens are registered and configured, everything works at this point.
3. I added to the config services: mailTemplateListApiHandler: Factory: Crm\RempMailerModule\Api\MailTemplateListApiHandler setup:
   • addAllowedMailTypeCodes('new_subscribers')

and then I can add templates with this key to Mailer.

4.Hmm do I understand correctly that this will solve the token issue for Mailer, and I can use SSO for other modules (Beam, Segments, etc.) But Mailer will no longer send token verification to SSO module?

Also Peter Your help is very important to us. I really appreciate it. I have one more request for you! Please, I need help with the Beam installation process. If it is possible, please write me a "Hello" to the email address indicated in my profile so that I don't create a bunch of issues. Thank you very much!