Open Codelica opened 4 years ago
Have a look at https://www.github.com/tiredofit/docker-debian/ for the settings you can use for Email - There are a series of SMTP_ environment variables that allow you to send messages out to an external server. It does not send directly to a recipient however..
Thanks... I configured the env vars and can get a successful email from inside at the shell via msmtp -f freepbx@mydomain.com me@mydomain.com
but for some reason they aren't coming out from FreePBX backup. The error went away, so I'm thinking they are in limbo :) hunting for logs and clues. Thanks for the pointer though !
OK, so good the container is working in terms of mailing out. It may actually be in the Freepbx code where they have hardcoded something. In earlier revisions there was quite a lot of hardcoding that we had to work around. I'll see what I can do at scanning through their code for a pointer.
Looking through here at their email classes: https://github.com/FreePBX/framework/blob/3b2fdebbe1c7fb7b54606c68efb36ec4574c923e/amp_conf/htdocs/admin/libraries/BMO/Mail.class.php it seems they have hardcoded a localhost and 25 unless the Sysadmin Module is installed. That may or may not be the case. I do recall having at least some components letting me know when something had occurred via email. I suppose we'd just need to figure out what value the sysadmin plugin is writing to what table and we could certainly work around it.
Ahh.. nice spot! That would be great, but don't burn too much time. :) To test things out I just added a proxy on localhost:25 with socat inside and the mail delivered fine.
socat tcp-listen:25,reuseaddr,fork tcp:my-mailserver.com:25
I reported about this situation: https://issues.freepbx.org/browse/FREEPBX-21595 proxy is good idea, I will make PR for this.
Update : I've made a couple of steps... After setting ENABLE_SMTP to true, as noted in another comment, I got this:
s6-supervise (child): fatal: unable to exec run: Permission denied,
s6-supervise 15-socat: warning: unable to spawn ./run - waiting 10 seconds
I shelled into the running image, and ran ls -al /etc/services.d/15-socat/run
No user had execute set! So I ran chmod a+x /etc/services.d/15-socat/run
The image then finished booting up. I also set SMTP_HOST to my on LAN SMTP relay (namshi/smtp image)
Next problem : I can send emails by writing a test file and catting it to msmtp, but I have to use:
msmtp -f some@address.here
Else it returns:
msmtp: account default from /etc/msmtprc: envelope-from address is missing
So I know the relay via socat is working, and my relay host config is ok - but whenever I generate a voicemail, I don't get an email...email is enabled, my address is valid - so I presume the msmtp is arguing.
There is no from address configured in /etc/msmtprc currently. Even changing the freepbx mailcmd in voicemail admin to /usr/sbin/sendmail -t -f valid@address.com doesn't generate an email....but I can't see an error trapped anywhere
Looking in /var/log/asterisk/full (with debug on) I do see this after hanging up on sending a voicemail
[2020-07-27 11:35:53] DEBUG[6038][C-00000008] app_voicemail.c: Sent mail to my.address@gmail.com with command '/usr/bin/sendmail -t -f myvalidfromaddress@mydomain.com'
msmtp will send to/from those addresses manually fine - and (manually) I see the smtp connection on my smtp relay container - but when Asterisk/FreePBX says it's sending, nothing hits the Container, and thus (I presume) not even socat.
A couple weeks ago the socat service was brought in to solve some hardcoded quirks with the backup module. I'll push a new copy of freepbx immediately with the +x bit on for that service. I'm trying to think back at what I had to do to get voicemail emails going - it's been a while. I use tiredofit/postfix-relay for my outbound smtp (even to a local LAN based SMTP) but what you are using should be fine as well. I'll do some digging in the asterisk source to see if something has changed. I'll also investigate this FROM-Address error. I did notice it on one of my other images lately so looks like we'll have to add another var.
In the meantime, there is an documented var called SMTP_ENABLE_GMAIL which adds the entry auto_from on
which may give you better results at least from the command line.
Will push this new image with working socat right away and then determine what we need to do with MSMTP following that.
Hmm, not sure what's going on here. Last night it managed to send me an email about upgradable modules, so that's something. Voicemail notifications? Nope.
I noted it sent the email from asterisk@freepbxengine - freepbxengine is the hostname I gave the Docker Container, but I can't otherwise see where the address gets/got set? (unless it picked it up via the sendmail invocation as the calling user)
Right, cracked it (sort of) Everytime I run msmtp manually with just a rcpt address (/usr/bin/sendmail me@mydomain.com) I got an error about the envelope-from address missing from /etc/mstmprc
So I updated my /etc/mstmprc thus:
_### Automatically generated on container start. See documentation on how to set! account default from my-sender@mydomain.com host 192.168.1.50 port 25 domain docker maildomain local tls off tls_starttls off tlscertcheck off
...and then I changed the freepbx voicemail send command to this:
/usr/sbin/sendmail -t
And tested, and it worked! So, either this may need to be an ENV, or (if it's set) picked up from the vm config itself (there is a server email box - although typically it's blank until set.
Addendum - if you could add "apt-get update" and "apt-get install dos2unix" to the package I'd be immensely grateful (I have a custom email to mp3 script which otherwise runs, but uses dos2unix) :)
Happy to share if interested, as it calls IBM Watson audio to text for translation. All other utils it needs are present, and if you provided one (or two) additional ENV settings, it could be included in image as an option.
Thanks Peter. Normally an application would send the From header as well but as we've seen with Freepbx it's skipping a few key functions and uses different methods for different areas.
I'd be more than happy to include it in the package if you wanted to submit it as a PR. We can certainly add environment variables to be modified upon container bootup as well.
So what I'll do today is update the Debian base image with the from statement in msmtprc (env var). It'll take about 3 days for Docker Hub to finish off building all the dependent images on it, but we should have solved this email issue right after. I'll also put in dos2unix for you in the meantime.
I've not forked/pulled your codebase to generate a PR. For the MP3/transcription It's a single file, so I could paste a marked up version in this chain?
Sure that works! Debian base images are building, let's cross our fingers Docker Hub builds quick :)
Ok - here is sendmailmp3.sh This can live anywhere as long as users know the path - so they can set the FreePBX VM Sendmail cmd path to it ATM I'm using a copy in my mounted /data folder
ENV vars I've set
VM_TRANSCRIBE - can be TRUE or FALSE (basically only does stuff if set to TRUE - suggest default of FALSE if not set VM_TRANSCRIBE_APIKEY - IBM Watson API Key (URL comments inline - an account is needed) - should not be blank VM_TRANSCRIBE_MODEL - see comments in line for valid values URL - suggest default of en-GB_NarrowbandModel
#!/bin/sh
# Asterisk voicemail attachment conversion script
# Revision history :
# 22/11/2010 - V1.0 - Creation by N. Bernaerts
# 07/02/2012 - V1.1 - Add handling of mails without attachment (thanks to Paul Thompson)
# 01/05/2012 - V1.2 - Use mktemp, pushd & popd
# 08/05/2012 - V1.3 - Change mp3 compression to CBR to solve some smartphone compatibility (thanks to Luca Mancino)
# 01/08/2012 - V1.4 - Add PATH definition to avoid any problem (thanks to Christopher Wolff)
# 16/07/2015 - V1.5 - Handle natively GSM WAV (thanks to Michael Munger)
# 01/08/2020 - V2.0 - Modded for revised IBM Watson URLs and Docker tiredofit/docker-freepbx
# Requires
# dos2unix
# sed
# awk
# lame
# set PATH
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
# create a temporary directory and cd to it
TMPDIR=$(mktemp -d)
cd $TMPDIR
# dump the stream to a temporary file
cat >> stream.org
# get the boundary
BOUNDARY=$(grep "boundary=" stream.org | cut -d'"' -f 2)
# cut the file into parts
# stream.part - header before the boundary
# stream.part1 - header after the bounday
# stream.part2 - body of the message
# stream.part3 - attachment in base64 (WAV file)
# stream.part4 - footer of the message
awk '/'$BOUNDARY'/{i++}{print > "stream.part"i}' stream.org
# if mail is having no audio attachment (plain text)
PLAINTEXT=$(cat stream.part1 | grep 'plain')
if [ "$PLAINTEXT" != "" ]
then
# prepare to send the original stream
cat stream.org > stream.new
# else, if mail is having audio attachment
else
# cut the attachment into parts
# stream.part3.head - header of attachment
# stream.part3.wav.base64 - wav file of attachment (encoded base64)
sed '7,$d' stream.part3 > stream.part3.wav.head
sed '1,6d' stream.part3 > stream.part3.wav.base64
# convert the base64 file to a wav file
dos2unix -o stream.part3.wav.base64
base64 -di stream.part3.wav.base64 > stream.part3.wav
# convert wave file (GSM encoded or not) to PCM wav file
sox stream.part3.wav stream.part3-pcm.wav
if [ "$VM_TRANSCRIBE" = "TRUE" ]
then
CURL_OPTS=""
# Do not change the API_USERNAME
API_USERNAME="apikey"
# Set the API_PASSWORD to relevant $ENV variable
# IBM Watson PAYG account with speech transcription enabled - is required
# More info at https://www.ibm.com/uk-en/cloud/watson-speech-to-text
# See also http://nerdvittles.com/?page_id=25616 for more info on how to setup the IBM account
API_PASSWORD="$VM_TRANSCRIBE_APIKEY"
# Set the voice model to a valid IBM model
# All models available via https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models
API_MODEL=$VM_TRANSCRIBE_MODEL
curl -s $CURL_OPTS -k -u $API_USERNAME:$API_PASSWORD -X POST \
--limit-rate 40000 \
--header "Content-Type: audio/wav" \
--data-binary @stream.part3.wav \
"https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true&model=$VM_TRANSCRIBE_MODEL" 1>audio.txt
# Extract transcript results from JSON response
TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "$##g'`
fi
# convert PCM wav file to mp3 file
# -b 24 is using CBR, giving better compatibility on smartphones (you can use -b 32 to increase quality)
# -V 2 is using VBR, a good compromise between quality and size for voice audio files
lame -m m -b 24 stream.part3-pcm.wav stream.part3.mp3
# convert back mp3 to base64 file
base64 stream.part3.mp3 > stream.part3.mp3.base64
# generate the new mp3 attachment header
# change Type: audio/x-wav or audio/x-WAV to Type: audio/mpeg
# change name="msg----.wav" or name="msg----.WAV" to name="msg----.mp3"
sed 's/x-[wW][aA][vV]/mpeg/g' stream.part3.wav.head | sed 's/.[wW][aA][vV]/.mp3/g' > stream.part3.mp3.head
# generate first part of mail body, converting it to LF only
mv stream.part stream.new
cat stream.part1 >> stream.new
cat stream.part2 >> stream.new
if [ "$VM_TRANSCRIBE" = "TRUE" ]
then
echo "--- Automated transcription result ---" >> stream.new
echo "$TRANSCRIPT" >> stream.new
fi
cat stream.part3.mp3.head >> stream.new
dos2unix -o stream.new
# append base64 mp3 to mail body, keeping CRLF
unix2dos -o stream.part3.mp3.base64
cat stream.part3.mp3.base64 >> stream.new
# append end of mail body, converting it to LF only
echo "" >> stream.tmp
echo "" >> stream.tmp
cat stream.part4 >> stream.tmp
dos2unix -o stream.tmp
cat stream.tmp >> stream.new
fi
# send the mail thru sendmail
cat stream.new | sendmail -t
# remove all temporary files and temporary directory
rm -Rf $TMPDIR
Alright, I've made a few minor mods to the script and put it in place inside the source tree. I'm going to wait for a couple hours before I commit to make sure Docker Hub doesn't go out of sequence but we should see this tagged as release 15-5.0.4
in the next 36 hours.
Excellent stuff. Did you set an ENV for the msmtprc?
Yes, that's in the top level base image of tiredofit/debian now, with the variable SMTP_FROM
which is where the holdup on pushing this new image out - Docker Hub only gives me 1 core to build with and when working at the very top level of my images it has to walk through every dependent image and rebuild. On my private build server it takes approximately 7 hours on a 16 core system, but that is also including the Alpine variants which most of my images are using, so it's a waiting game at this stage.
Image is live now. Success on the mail sending issue?
Downloading 15.5.0.4 and 15:latest to check. Findings on latest: 1) I have set SMTP_FROM in env and it's not picked up in /etc/msmtprc (no from line - I've manually put it back) 2) dos2unix is present 3) Checking startup logs I can see the following two lines: ++ ENABLE_VM_TRANSCRIBE=FALSE, ++ VM_TRANSCRIBE_MODEL=en-GB_NarrowbandModel,
but no reference to SMTP_FROM (1) or VM_APIKEY?
Can you confirm the name of the script when you added to source?
I ranfind / -name *send*.sh
and can't find anything but my own mounted ones?
Updated (few days off) I can see the watson_transcribe script, but it's not working correctly with all ENVs set. Voicemail received but not transcription. Also, I set FOP to be enabled and it won't autostart. A manual kick of /usr/local/fop2/fop2_server -D gets it going
Weird - And it is working when you manually inject? Are we positive that it is executing? A simple check would be to add a
echo "Script executed at $(date)" >> /tmp/watson.log
line on line 2 after the #!/bin/bash declaration.
I'll check in a mo - just having the normal fun of a reboot and all my containers firing out of order.... Also, forgot to say /etc/msmptrc didn't pick up a from line (so I put that back in manually also)
Update: Script is called but the transcribing isn't working - will see what I can see.
So, my script (which lives outside sbin) is leaving a folder in /tmp owned by asterisk:asterisk Your script (in sbin) is leaving a folder owned by root:root - which I suspect asterisk can't read/write into ( 1 x 0 byte file in it)
The sbin/ownership may be the issue - I forced the /usr/sbin copy to be chown asterisk:asterisk and I'm getting files in the folder, but the audio.txt file is missing (which is called by Curl to IBM) - so checking that now
Weird - SMTP_FROM didn't make it into the buster source tree yet I explicitly added it. I'm pushing a new release of the base image now for it.
Right, found the issues with the watson_transcribe a) you def need to chown asterisk:asterisk the script in /usr/sbin b) change the line in the script that reads:
curl -s "$CURL_OPTS" -k -u $API_USERNAME:"$API_PASSWORD" -X POST \
to
curl -s "$CURL_OPTS" -k -u $API_USERNAME:$API_PASSWORD -X POST \
(lose the quotes around API_PASSWORD - as they are passed to IBM which bounced them.
Otherwise it's calling and creating audio.txt fine, which should then get wrapped into the mail (but isn't - so I'm looking at that too)
Have a peek at this updated script which if you have the environment variable CONTAINER_LOG_LEVEL=DEBUG
set will step through the script:
#!/usr/bin/with-contenv bash
# Asterisk voicemail attachment conversion script
# Revision history :
# 22/11/2010 - V1.0 - Creation by N. Bernaerts
# 07/02/2012 - V1.1 - Add handling of mails without attachment (thanks to Paul Thompson)
# 01/05/2012 - V1.2 - Use mktemp, pushd & popd
# 08/05/2012 - V1.3 - Change mp3 compression to CBR to solve some smartphone compatibility (thanks to Luca Mancino)
# 01/08/2012 - V1.4 - Add PATH definition to avoid any problem (thanks to Christopher Wolff)
# 16/07/2015 - V1.5 - Handle natively GSM WAV (thanks to Michael Munger)
# 01/08/2020 - V2.0 - Modded for revised IBM Watson URLs and Docker tiredofit/docker-freepbx
# 01/08/2020 - V2.1 - Further tweaks to integrate within tiredofit/docker-freepbx
# Requires
# dos2unix
# sed
# awk
# lame
# Grab functions and defaults from Docker Image
source /assets/functions/00-container
source /assets/defaults/10-freepbx
print_debug "Set Path"
# set PATH
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
print_debug "Create Temporary Directory"
# create a temporary directory and cd to it
TMPDIR=$(mktemp -d)
cd "$TMPDIR"
print debug "Now working in ${TMPDIR}"
# dump the stream to a temporary file
print_debug "Dumping Stream to Temporary File"
cat >> stream.org
print_debug "Temporary file is $(du -k "$stream.org" | cut -f 1) kilobytes
"
print_debug "Getting Boundaries"
# get the boundary
BOUNDARY=$(grep "boundary=" stream.org | cut -d'"' -f 2)
# cut the file into parts
# stream.part - header before the boundary
# stream.part1 - header after the bounday
# stream.part2 - body of the message
# stream.part3 - attachment in base64 (WAV file)
# stream.part4 - footer of the message
print_debug "Cutting file into parts"
awk '/'$BOUNDARY'/{i++}{print > "stream.part"i}' stream.org
# if mail is having no audio attachment (plain text)
PLAINTEXT=$(cat stream.part1 | grep 'plain')
if [ "$PLAINTEXT" != "" ] ; then
print_debug "File has no audio attachment"
# prepare to send the original stream
cat stream.org > stream.new
print_debug "Sending to new Stream"
# else, if mail is having audio attachment
else
print_debug "File has audio attachment"
# cut the attachment into parts
# stream.part3.head - header of attachment
# stream.part3.wav.base64 - wav file of attachment (encoded base64)
print_debug "Cutting Attachment into parts"
sed '7,$d' stream.part3 > stream.part3.wav.head
sed '1,6d' stream.part3 > stream.part3.wav.base64
# convert the base64 file to a wav file
print_debug "Converting Base64 information into WAV file"
dos2unix -o stream.part3.wav.base64
base64 -di stream.part3.wav.base64 > stream.part3.wav
# convert wave file (GSM encoded or not) to PCM wav file
print_debug "Convert WAV file to PCM"
sox stream.part3.wav stream.part3-pcm.wav
if var_true "$ENABLE_VM_TRANSCRIBE" ; then
print_debug "Voicemail Transcription Enabled"
CURL_OPTS=""
# Do not change the API_USERNAME
API_USERNAME="apikey"
# Set the API_PASSWORD to relevant $ENV variable
# IBM Watson PAYG account with speech transcription enabled - is required
# More info at https://www.ibm.com/uk-en/cloud/watson-speech-to-text
# See also http://nerdvittles.com/?page_id=25616 for more info on how to setup the IBM account
API_PASSWORD="$VM_TRANSCRIBE_APIKEY"
# Set the voice model to a valid IBM model
# All models available via https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models
API_MODEL=$VM_TRANSCRIBE_MODEL
print_debug "Using API_MODEL ${VM_TRANSCRIBE_MODEL}"
print_debug "Sending Payload to Watson"
curl -s "$CURL_OPTS" -k -u $API_USERNAME:$API_PASSWORD -X POST \
--limit-rate 40000 \
--header "Content-Type: audio/wav" \
--data-binary @stream.part3.wav \
"https://stream.watsonplatform.net/speech-to-text/api/v1/recognize?continuous=true&model=$VM_TRANSCRIBE_MODEL" 1>audio.txt
print_debug "Extracting Transcript Results"
# Extract transcript results from JSON response
TRANSCRIPT=$(cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "$##g')
fi
# convert PCM wav file to mp3 file
# -b 24 is using CBR, giving better compatibility on smartphones (you can use -b 32 to increase quality)
# -V 2 is using VBR, a good compromise between quality and size for voice audio files
print_debug "Converting WAV to MP3"
lame -m m -b 24 stream.part3-pcm.wav stream.part3.mp3
# convert back mp3 to base64 file
print_debug "Converting MP3 to Base64"
base64 stream.part3.mp3 > stream.part3.mp3.base64
# generate the new mp3 attachment header
# change Type: audio/x-wav or audio/x-WAV to Type: audio/mpeg
# change name="msg----.wav" or name="msg----.WAV" to name="msg----.mp3"
print_debug "Creating Mp3 Attachment Header"
sed 's/x-[wW][aA][vV]/mpeg/g' stream.part3.wav.head | sed 's/.[wW][aA][vV]/.mp3/g' > stream.part3.mp3.head
# generate first part of mail body, converting it to LF only
print_debug "Generate 1st part of mail body"
mv stream.part stream.new
cat stream.part1 >> stream.new
cat stream.part2 >> stream.new
if var_true "${VM_TRANSCRIBE}" ; then
print_debug "Showing Transcribe Details"
echo "--- Automated transcription result ---" >> stream.new
echo "$TRANSCRIPT" >> stream.new
fi
print_debug "Generate third part of mail body"
cat stream.part3.mp3.head >> stream.new
dos2unix -o stream.new
print_debug "Appending Base64 MP3 to mail body"
# append base64 mp3 to mail body, keeping CRLF
unix2dos -o stream.part3.mp3.base64
cat stream.part3.mp3.base64 >> stream.new
# append end of mail body, converting it to LF only
print_debug "Append end of mail body"
echo "" >> stream.tmp
echo "" >> stream.tmp
cat stream.part4 >> stream.tmp
dos2unix -o stream.tmp
cat stream.tmp >> stream.new
fi
# send the mail thru sendmail
print_debug "Sending stream via sendmail"
cat stream.new | sendmail -t
# remove all temporary files and temporary directory
print_debug "Cleaning up"
rm -rf "$TMPDIR"
This is a dual issue :) Notwithstanding the permission/quote issue - my docker had lost the VM_ENABLE_TRANSCRIBE option....restarting...
OK. let me know. I'm going to bypass Docker Hub auto builds and build my own copy and can push new versions immediately to get you working with both SMTP_FROM and the transcribe script and FOP2.
Try ENABLE_VM_TRANSCRIBE
for your env var.
That one was my bad - changed. Found another issue in the script
The first TRANSCRIBE check is:
if var_true "$ENABLE_VM_TRANSCRIBE" ; then
but lower down, there is
if var_true "${VM_TRANSCRIBE}" ; then
print_debug "Showing Transcribe Details"
echo "--- Automated transcription result ---" >> stream.new
echo "$TRANSCRIPT" >> stream.new
fi
That does the append and has no matching value :) I've updated my local copy and it's appending So, change chown, remove quotes, amend variable - now it's working.
Ack I missed it. Sorry about that.
I'm building a new version and will let you know when it is in place. Fixed - Script based on our discussions SMTP_FROM FOP2 startup
Coolio - and no worries, not your script originally, and it's been pinned down :) I'm running the Chrome FOP plugin so that'a a handy canary if FOP doesn't start as it has a scrolling error text that vanishes once it signs on
On Sat, 15 Aug 2020 at 15:29, Dave Conroy notifications@github.com wrote:
I'm building a new version and will let you know when it is in place. Fixed - Script based on our discussions SMTP_FROM FOP2 startup
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tiredofit/docker-freepbx/issues/146#issuecomment-674405161, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPGIY7Y3EKPZYN2FF7OOS3SA2LVRANCNFSM4N2UEWUA .
My locally built copy uploading now to Docker Hub as tiredofit/freepbx:15-latest
Docker Hub should follow with it's own tagged images of version 5.0.6
.
First off thanks for this image. Saved me a TON of time having to quickly migrate to FPBX 15 and PJSIP. :)
Everything is working well except for one snag, email. When trying to send I get errors like the following:
I saw a prior closed issue about msmtp being in the base image and can see it's included, however it doesn't seem to be providing service. I've tried explicitly adding
ENABLE_SMTP=TRUE
but no go. I figure I'm missing something simple. Any pointers?Thanks!