search

HackerExperience / Helix

GNU Affero General Public License v3.0
53 stars 10 forks source link

Rewrite TOP #322

Closed renatomassaro closed 6 years ago

renatomassaro commented 6 years ago

Depends on #317. Closes #318. Closes #291. Closes #262. Closes #215. Closes #199.

  Here are a few factors that TOP must consider when Allocating resources:

  1. A process may use all, one or zero resources from a server. Resources that
  a process does not use should never be allocated to it. Instead, these
  resources must be available to other processes who actually need them.

  2. A process may have a minimum amount of resources it uses. This is called
  `static`, later on referred to as the "Static Allocation" of a process.

  3. A process may have a maximum amount of resources it may use. This is
  referred to as the `limit` of a process.

  4. A process may consume all available resources on the server (as long as it
  needs that resource, see 1). This is referred as "Dynamic Allocation" of a
  process. Dynamic because it depends on how many server resources are available

  5. A process may be paused. A paused process does not consume available server
  resources (dynamic) but it may consume a minimum amount of resources (static).
  As such, the `static` resource consumption of a process exists for both when
  it is running, and when it is paused, and it may have different values.

  6. A process may have a priority relative to other processes on the same
  server. A process with higher priority receives more resource shares than
  others with lower priority.

  7. A process allocation may change if more or less server resources are made
  available (e.g. in the case another process has completed or started). When
  this happens, the process needs to have its allocation recalculated, taking
  in consideration the new amount of server resources.

  8. The sum of all processes allocation must not exceed the available resources
  on a server, in which case we have a resource overflow scenario.

  9. When a process allocation changes, we need to store somewhere how many
  resources it had processed before, with the previous amount allocated.

  10. A process may exist on two different servers, referred to as `local` and
  `remote` (or with the prefix `l_` and `r_`, respectively).

  11. Because of 10, the total resources allocated on a process may be limited
  by the remote server's resources (e.g. a FileTransferProcess may be limited by
  the remote server's ULK or DLK.).

  12. Similar to 7, if more resources are made available on the remote server,
  we must recalculate the process allocation on both the local and the remote
  servers, since now the process may be allowed to allocate more or less
  resources.

  13. In fact, a process' limit and dynamic consumption, described on items 3
  and 4 respectively, also apply to the remote server.

  (In order to keep the mental sanity of Helix developers, we've postponed the
  option for remote processes to have their own `objective`, `processed` and
  `static` allocation. But it's doable, and probably needed for MalwareProcess).

  14. When the total resources available in a server changes, this may affect
  all processes that target/originate from this server (12). When recalculating
  the resources used on these other processes, it may be the case that *their*
  recalculations would affect processes that target/originate *them*, in which
  case they should be recalculated too. And so on. Recursively.

  15. Oh, it must be fast - less than 1ms. And accurate - millisecond accurate.
  And simple to understand.

TODO:

Incidental

This change is Reviewable

sourcelevel-bot[bot] commented 6 years ago

Ebert has finished reviewing this Pull Request and has found:

You can see more details about this review at https://ebertapp.io/github/HackerExperience/Helix/pulls/322.

renatomassaro commented 6 years ago

:top:

Reviewed 173 of 173 files at r1. Review status: all files reviewed at latest revision, 9 unresolved discussions.

lib/event/meta.ex, line 37 at r1 (raw file):

    :process_id,

    :stack

doc

lib/process/executable.ex, line 277 at r1 (raw file):

              {:error, :internal}

            _ ->

nope

lib/process/action/flow/process.ex, line 2 at r1 (raw file):

defmodule Helix.Process.Action.Flow.Process do

doc

lib/process/event/top.ex, line 2 at r1 (raw file):

defmodule Helix.Process.Event.TOP do

doc

lib/process/event/top.ex, line 9 at r1 (raw file):

    alias Helix.Process.Model.Process

    @type t :: term

type

lib/process/model/process/resources.ex, line 1 at r1 (raw file):

defmodule Helix.Process.Model.Process.Resources.Utils do

move another file

lib/process/model/process/resources.ex, line 15 at r1 (raw file):

import Helix.Process.Resources

resources Helix.Process.Model.Process.Resources do

doc module

lib/process/query/top.ex, line 10 at r1 (raw file):

  @spec load_top_resources(Server.idt) ::
    Process.Resources.t
  def load_top_resources(server = %Server{}) do

doc

lib/process/resources/behaviour.ex, line 12 at r1 (raw file):

  @type resource :: term

  @type process :: term

type

Comments from Reviewable

renatomassaro commented 6 years ago

Reviewed 16 of 16 files at r2. Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable