Realize Distributed Architecture

[Kyeg]

To make our system become Distributed Architecture (i.e. increasing elasticity, resilience, scalability...), there are something Soumya has taught in class

• [ ] (1) Concurrency Asynchronicity : Multiple I/O Tasks
• [x] (2) Parallelism Asynchronicity : Distributed Queue Service & Background Worker
  • [x] (a) Distributed Queue Service
  • [ ] (b) On-demand Background Worker
  • [x] (c) Scheduled Background Worker
• [ ] (3) Responsive : Use Web Sockets & HTTP post to connect browser & background worker
• [x] (4) Distributed Caching/Stubbing & Performance Benchmark: Browser Caching, Reverse-proxy Caching, Proxy Caching
• [x] (5) Distributed Production Environment :
  • [x] (a) Deployment Platform like Heroku
  • [ ] (b) Automated CI/CD like GitHub Action
  • [ ] (c) Dockerized Deployment

In the requirements for the final presentations, we need to apply AT LEAST ONE of above techniques to improve our system, especially AT LEAST ONE of Parallelism skills.

[Kyeg]

I will try to add these techniques in this order: (5-a) -> (2-a) -> (2-c)

[Kyeg]

If we still have time, we can add some additional techniques, and because Soumya emphasizes the Concurrency & Parallelism more, I guess a reasonable order maybe: (1) -> (2-b) -> (3) -> (4) -> (5-b) & (5-c)

[derrxb]

Hi, how is this going? Some feedback would be appreciated as today is the last day we can really write code. Soumya mentioned that we only need to choose one: concurrency or a scheduled worker. As long as we have an intensive task in our project.

[Kyeg]

I am now working on Scheduled Background Worker: i.e. (5-a) -> (2-a) -> (2-c) Before I complete them and create a PR, I will keep updating the following checkbox.

• [x] Heroku Deployment
  • Set Heroku Environment
  • Create Heroku Cloud App (in the root repository of the web app)
  • Set Postgresql DB
  • Set Production Environment
  • Set Procfile
  • Operate Heroku Cloud App
• [x] Configure AWS Account & SQS
  • Add IAM user & Add Permission
  • Rakefile: queue (create new queue)
• [x] API Server
  • Gemfile: gem 'aws-sdk-sqs'
  • Rakefile: queue & worker
  • Config: secrets.yml (AWS Configuration, Queue Configuration)
  • GitHub Action Workflows: api.yml (AWS Configuration, Queue Configuration)
  • Infrastructure/Messaging/: queue.rb (init, send, poll...)
  • Application/Services/: notify_workers in specific service (i.e. Publish)
• [x] Scheduled Background Worker
  • Gemfile: gem 'aws-sdk-sqs'
  • Infrastructure/Messaging/: queue.rb (init, send, poll...)
  • Worker/Application/: xxx_scheduled_worker.rb (i.e. Polling)
  • Reuse the Infrastructure/ & Representers/
• [ ] Scheduler on Heroku
  • $ heroku addons:create scheduler
  • $ heroku addons:open scheduler
    • Setup Scheduler on Web: Command to run / Frequency of runs
  • $ heroku logs --tail

[derrxb]

Awesome! I can't wait to see the end product. In the meantime, I will continue working on adding the features that we need for the app. Right now I'm doing the subscriptions and then after I'm going to work on the favorites. 

[Kyeg]

When I try to deploy the latest app on the Heroku. There is some error: Econfig::NotFound: configuration key 'REDISCLOUD_URL' is not set

@derrxb Do you change the secret.yml or something else about production?

[derrxb]

Nope, secrets.yml wouldn't have an effect either since on production we use heroku's config variables: https://dashboard.heroku.com/apps/api-mindmap/settings.

[Kyeg]

After some configuration, I can now deploy both the API & APP, but the results look different between the Heroku and the local test.

Now the API is running at: https://mytest-api-mind-map.herokuapp.com/ And the APP is running at: https://mytest-app-mind-map.herokuapp.com/

[Kyeg]

In the dev mode, wrong inbox id wont break the app:

[Kyeg]

However, in the Heroku App, wrong inbox id will break the app:

[Kyeg]

And the error message in the App server looks like this:

2021-01-12T14:27:13.818924+00:00 heroku[router]: at=info method=GET path="/inbox?inbox_id=abc" host=mytest-app-mind-map.herokuapp.com request_id=6e0a76f0-bfd9-46f7-9636-5dc741dc6510 fwd="123.192.83.159" dyno=web.1 connect=0ms service=1ms status=302 bytes=109 protocol=https
2021-01-12T14:27:14.134806+00:00 heroku[router]: at=info method=GET path="/inbox/abc" host=mytest-app-mind-map.herokuapp.com request_id=2cf295e5-38be-4450-9012-680c2508ece3 fwd="123.192.83.159" dyno=web.1 connect=0ms service=1ms status=500 bytes=154 protocol=https
2021-01-12T14:27:14.134136+00:00 app[web.1]: 2021-01-12 14:27:14 +0000: Rack app error handling request { GET /inbox/abc }
2021-01-12T14:27:14.134147+00:00 app[web.1]: #<TypeError: no implicit conversion of Array into String>
2021-01-12T14:27:14.134148+00:00 app[web.1]: /app/app/application/services/get_inbox.rb:20:in `+'
2021-01-12T14:27:14.134148+00:00 app[web.1]: /app/app/application/services/get_inbox.rb:20:in `rescue in request_inbox'
2021-01-12T14:27:14.134148+00:00 app[web.1]: /app/app/application/services/get_inbox.rb:15:in `request_inbox'
2021-01-12T14:27:14.134149+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/callable.rb:33:in `call'
2021-01-12T14:27:14.134149+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/callable.rb:33:in `call'
2021-01-12T14:27:14.134172+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step_adapters/around.rb:12:in `call'
2021-01-12T14:27:14.134173+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step_adapters/raw.rb:11:in `call'
2021-01-12T14:27:14.134174+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step_adapter.rb:41:in `call'
2021-01-12T14:27:14.134174+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step_adapter.rb:41:in `call'
2021-01-12T14:27:14.134175+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step.rb:54:in `block in call'
2021-01-12T14:27:14.134175+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step.rb:61:in `with_broadcast'
2021-01-12T14:27:14.134175+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/step.rb:54:in `call'
2021-01-12T14:27:14.134176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/stack.rb:19:in `block (3 levels) in compile'
2021-01-12T14:27:14.134176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-monads-1.3.5/lib/dry/monads/right_biased.rb:57:in `bind'
2021-01-12T14:27:14.134177+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/stack.rb:19:in `block (2 levels) in compile'
2021-01-12T14:27:14.134177+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/stack.rb:12:in `call'
2021-01-12T14:27:14.134177+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/dry-transaction-0.13.0/lib/dry/transaction/instance_methods.rb:28:in `call'
2021-01-12T14:27:14.134178+00:00 app[web.1]: /app/app/application/controllers/app.rb:88:in `block (4 levels) in <class:App>'
2021-01-12T14:27:14.134178+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:499:in `always'
2021-01-12T14:27:14.134178+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:490:in `_verb'
2021-01-12T14:27:14.134178+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:93:in `get'
2021-01-12T14:27:14.134178+00:00 app[web.1]: /app/app/application/controllers/app.rb:87:in `block (3 levels) in <class:App>'
2021-01-12T14:27:14.134179+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:564:in `if_match'
2021-01-12T14:27:14.134179+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:211:in `on'
2021-01-12T14:27:14.134179+00:00 app[web.1]: /app/app/application/controllers/app.rb:53:in `block (2 levels) in <class:App>'
2021-01-12T14:27:14.134179+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:564:in `if_match'
2021-01-12T14:27:14.134180+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/request.rb:211:in `on'
2021-01-12T14:27:14.134180+00:00 app[web.1]: /app/app/application/controllers/app.rb:35:in `block in <class:App>'
2021-01-12T14:27:14.134180+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda.rb:486:in `block in _roda_handle_main_route'
2021-01-12T14:27:14.134180+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda.rb:484:in `catch'
2021-01-12T14:27:14.134181+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda.rb:484:in `_roda_handle_main_route'
2021-01-12T14:27:14.134181+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda/plugins/error_handler.rb:87:in `_roda_handle_main_route'
2021-01-12T14:27:14.134181+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/roda-3.39.0/lib/roda.rb:370:in `block in base_rack_app_callable'
2021-01-12T14:27:14.134245+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
2021-01-12T14:27:14.134246+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
2021-01-12T14:27:14.134246+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
2021-01-12T14:27:14.134246+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/puma-3.12.6/lib/puma/configuration.rb:227:in `call'
2021-01-12T14:27:14.134251+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/puma-3.12.6/lib/puma/server.rb:706:in `handle_request'
2021-01-12T14:27:14.134251+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/puma-3.12.6/lib/puma/server.rb:476:in `process_client'
2021-01-12T14:27:14.134251+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/puma-3.12.6/lib/puma/server.rb:334:in `block in run'
2021-01-12T14:27:14.134251+00:00 app[web.1]: /app/vendor/bundle/ruby/2.7.0/gems/puma-3.12.6/lib/puma/thread_pool.rb:135:in `block in spawn_thread'

[Kyeg]

Any idea why https://mytest-api-mind-map.herokuapp.com/ & https://mytest-app-mind-map.herokuapp.com/ have different results with dev mode?

[Kyeg]

Ok, I think I have solved the problem. For production environment, some configuration in our code should be modified. I will push another PR afterwards.

[derrxb]

Sorry for not being able to provide more assistance with this. I've been working on getting the subscriptions front-end working. Were you able to fix the issue? Maybe you're calling the incorrect api path? I've received this error #<TypeError: no implicit conversion of Array into String> whenever the Front-end is not calling the backend correctly. Either because the api path is incorrrect of the api is not live/active