use self-hosted IMAP server for testing

[tomholub]

@DenBond7 can you please copy the whole test server setup from Android repo, and copy all IMAP test resources here as well? Then we can start writing test cases against it on iOS.

[tomholub]

The goal of this issue is that it will start and run in each Semaphore CI run like on Android, but remain unused.

Then @Kharchevskyi will switch iOS test to start using it in another PR.

[tomholub]

[DenBond7]

@Kharchevskyi Please feel free to ask me directly via skype. I will show you what I have. It'll be much faster.

[tomholub]

Actually I prefer to keep the communication here. It's very important for future reference. Communication in skype gets lost.

[DenBond7]

https://github.com/FlowCrypt/flowcrypt-android/blob/4d8f1ded212ccaff5f6f0aef7023038354c37c9d/FlowCrypt/src/androidTest/java/com/flowcrypt/email/ui/activity/MessageDetailsActivityTest.kt#L184

[tomholub]

@DenBond7 Regarding the source for the docker image that we use for IMAP/SMTP - where exactly is the source now? Is it on Android repo? Could you share a link?

[DenBond7]

We store the email server code here https://github.com/FlowCrypt/flowcrypt-android/tree/master/docker-mailserver

[tomholub]

Understood - is it also on dockerhub somewhere? Or you run docker just from that directory, when running tests on Android?

[DenBond7]

https://hub.docker.com/r/flowcrypt/flowcrypt-email-server

We've created an organization for that

[tomholub]

Great. In that case, for now, you can leave the sources in Android repo. Then on iOS, you can just use the dockerhub directly, without need for android repo - correct?

[tomholub]

Then moving https://github.com/FlowCrypt/flowcrypt-android/tree/master/docker-mailserver to a separate github repo can happen sometime in the future

[DenBond7]

Great. In that case, for now, you can leave the sources in Android repo. Then on iOS, you can just use the dockerhub directly, without need for android repo - correct?

Absolutely. I just need to add a few files to run all things via docker compose

[DenBond7]

@tomholub Could you give me access to Semaphore CI for this repo?

[tomholub]

Try now - you should have access.

[DenBond7]

Thank you! Now it works.

[DenBond7]

ref https://github.com/FlowCrypt/flowcrypt-ios/commit/77e5a242314a73edf54c8d14c5a74e1a7f90ec5f

[DenBond7]

@tomholub I've encountered that os_image: macos-xcode12 doesn't have installed docker. Also, I've not found an easy way to install docker via CLI.

@Kharchevskyi Could you investigate is it possible to install docker on Semaphore CI? I can't do it due to poor knowledge of the MacOS (actually I don't know any things about that).

[DenBond7]

@Kharchevskyi Anyway, you should be able to run an email server locally. I've already added the code.

to run the email server use the following command from the project root

cd docker-mailserver && ./run_email_server.sh

I just don't sure about 10.0.2.2. Maybe you should use localhost.

edit - added a text version

{
  "accountType": "flowcrypt.test",
  "areContactsLoaded": true,
  "email": "default@flowcrypt.test",
  "imapIsUseSslTls": true,
  "imapIsUseStarttls": false,
  "imapPort": 993,
  "imapServer": "10.0.2.2",
  "isActive": true,
  "isEnabled": true,
  "isRestoreAccessRequired": false,
  "isShowOnlyEncrypted": false,
  "password": "qwerty1234",
  "smtpIsUseSslTls": false,
  "smtpIsUseStarttls": true,
  "smtpPassword": "qwerty1234",
  "smtpPort": 587,
  "smtpServer": "10.0.2.2",
  "smtpUsername": "default@flowcrypt.test",
  "useCustomSignForSmtp": true,
  "username": "default@flowcrypt.test"
}

[Kharchevskyi]

@tomholub I've encountered that os_image: macos-xcode12 doesn't have installed docker. Also, I've not found an easy way to install docker via CLI.

@Kharchevskyi Could you investigate is it possible to install docker on Semaphore CI? I can't do it due to poor knowledge of the MacOS (actually I don't know any things about that).

• Make sure there is a Homebrew installed. Should be there based on https://docs.semaphoreci.com/ci-cd-environment/macos-xcode-12-image/ If not /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

• Install Docker brew install --cask docker

• Install docker compose brew install docker-compose

Will also try to run server locally on my Mac

[DenBond7]

@Kharchevskyi Thank you for the hints. I've tried but it doesn't work. You can use the debug tools to investigate that.

Please use the following commands and you will see results

denbond7@WorkPC:~$ sem debug job 5ааааааа....
* Creating debug session for job '5aaaaaa....
* Setting duration to 60 minutes
* Waiting for debug session to boot up ..
* Waiting for ssh daemon to become ready ..

Semaphore CI Debug Session.

  - Checkout your code with `checkout`
  - Run your CI commands with `source ~/commands.sh`
  - Leave the session with `exit`

Documentation: https://docs.semaphoreci.com/article/75-debugging-with-ssh-access.

semaphores-mac:~ semaphore$ checkout
Performing shallow clone with depth: 50
HEAD is now at 05c1666 Added missed settings.| #307
semaphores-mac:flowcrypt-ios semaphore$ brew install --cask docker
==> Downloading https://desktop.docker.com/mac/stable/amd64/63878/Docker.dmg
######################################################################## 100.0%
==> Installing Cask docker
==> Moving App 'Docker.app' to '/Applications/Docker.app'
🍺  docker was successfully installed!
semaphores-mac:flowcrypt-ios semaphore$ brew install docker-compose
....

semaphores-mac:~ semaphore$ source ~/commands.sh

[Kharchevskyi]

Couldn't manage to run it locally as well, will get in contact with @DenBond7 soonest possible

[github-actions[bot]]

There were 3 email addresses found in the above comment. Please:

1) click three dots -> edit to remove the email addresses 2) click edited in the comment header, and click on the previous revision of the comment 3) when viewing the old revision with an email in it, click options -> delete this revision from history

[Kharchevskyi]

@tomholub Managed to run locally. Thx @DenBond7 for the help Now will try to figure out is it possible to use if with iOS simulator, if yes will work on setting it on CI

[tomholub]

@Kharchevskyi I know you're currently working on the pass phrases in memory, but what was the status of this? Was there any issue with the iOS simulator calling this IMAP service?

[Kharchevskyi]

I could run server locally and there is a branch where was working or running it on CI.
https://flowcrypt.semaphoreci.com/branches/683cdcdc-e121-400e-9967-6df475fa3b03 I had some problems to connect to the server with simulator, so postponed a bit, will investigate later.

[tomholub]

We could at least merge the branch where the server starts up in background, with some added comments that it's not used yet - that's already some progress. Then we could look into hooking it up with the simulator later.

[tomholub]

I'll run the IMAP service on a separate server so we don't have to set it up in the CI.

Need to add this to /etc/hosts:

# flowcrypt-ios SMTP/IMAP server for CI
217.163.30.119 mail

When I SSH to the server and run this command, here's what I get:

# telnet mail 25
Trying 217.163.30.119...
Connected to mail.
Escape character is '^]'.
220 mail.flowcrypt.test ESMTP

But from my own device:

$ telnet mail 25
Trying 217.163.30.119...
Connected to mail.
Escape character is '^]'.

Likely a config issue on the SMTP server, will keep looking.

[tomholub]

@DenBond7 I'll send you ssh keys to the server by email. Sometime, please have a look how to configure the server so that it would accept connections from non-localhost clients.

[tomholub]

The key is here: https://github.com/FlowCrypt/flowcrypt-deploy/blob/master/dev/keys/ssh/dev-servers.key

$ ssh -i ~/git/flowcrypt-deploy/dev/keys/ssh/dev-servers.key root@217.163.30.119
# cd flowcrypt-ios/docker-mailserver

The service was started with docker-compose up -d mail.

[tomholub]

sorry, false alarm - it's a firewall issue

[tomholub]

I was able to run the imap server on a dedicated external ubuntu machine.

The goal of this issue has changed somewhat. Now that we have an external SMTP/IMAP running elsewhere, the app only needs to connect to it (during local dev cycle or during CI) to run tests.

1) add this to /etc/hosts

# flowcrypt-ios SMTP/IMAP server for CI
217.163.30.119 mail

2) try that you can connect cd docker-mailserver/ && ./check_email_server.sh

3) add IMAP user credentials to secrets file

4) separate tests in to Gmail tests and IMAP tests. IMAP tests will connect to mail:10025 and mail:10143 (non-TLS) SMTP/IMAP server during setup, and log in with one of the accounts from the secrets file, during tests

I'll soon close this issue and open another one.