search

openhpc / submissions

OpenHPC Component Submissions Project
8 stars 2 forks source link

Sensys #12

Closed harias closed 6 years ago

harias commented 7 years ago

Software Name

Sensys

Public URL

https://github.com/intel-ctrlsys/sensys

Technical Overview

Sensys provides resilient and scalable monitoring for resource utilization and node state of health, storing all collected data in a database for subsequent analysis. Sensys includes several loadable plugins that monitor various metrics related to different features present in each node like temperature, voltage, power usage, memory, disk and process information.

Complete documentation can be found here.

Latest stable version number

1.0

Open-source license type

BSD 3-Clause

Relationship to component?

If other, please describe:

Build system

If other, please describe:

Does the current build system support staged path installations? For example: make install DESTIR=/tmp/foo (or equivalent)

Does component run in user space or are administrative credentials required?

Does component require post-installation configuration.

If yes, please describe briefly: Out of the box Sensys can run on a single node collecting and processing monitoring data without any changes to the configuration, in hierarchical model, Sensys needs to be configured according to the cluster architecture and the available hardware. All configuration details can be found in the wiki's section Sensys User Guide.

If component is selected, are you willing and able to collaborate with OpenHPC maintainers during the integration process?

Does the component include test collateral (e.g. regression/verification tests) in the publicly shipped source?

The shipped source code includes unit tests based on Google Test framework. Some tests may require hardware and access to kernel modules to run.

The tests can be found on: https://github.com/intel-ctrlsys/sensys/tree/master/orcm/test

Does the component have additional software dependencies (beyond compilers/MPI) that are not part of standard Linux distributions?

Name Version
Sigar 1.6.5
libesmtp 1.0.6
net-snmp 5.7.3
net-snmp-devel 5.7.3
ipmiutil 2.9.6
ipmiutil-devel 2.9.6
postgresql 9.3
postgresql-devel 9.3
unixODBC 2.3.1
numactl-libs 2.0.9
openssl 1.0.1
zeromq 4.0.5
python 2.7
python-psycopg2 2.5.1
python-sqlalchemy 0.9.8
python-alembic 0.8.3

Does the component include online or installable documentation?

It is available as a Github Wiki page on https://github.com/intel-ctrlsys/sensys/wiki

[Optional]: Would you like to receive additional review feedback by email?

koomie commented 7 years ago

@harias, the list of required dependencies looks to include a number of items that are standard in the OS distros. Can you please clarify which dependencies are necessary that are not part of CentOS7 and SLES12?

Thanks.

harias commented 7 years ago

@koomie sure, the dependencies not included as part of the standard distros are:

Name Version
Sigar 1.6.5
ipmiutil 2.9.6
ipmiutil-devel 2.9.6
postgresql 9.3
postgresql-devel 9.3
zeromq 4.0.5
python-alembic 0.8.3

I can edit the issue to only show these dependencies. Thanks.

koomie commented 7 years ago

@harias, some of these that you list are available in the distros (e.g. postgresql, zeromq). By calling them out, are you saying that the distro versions are incompatible?

harias commented 7 years ago

@koomie the versions mentioned above are the ones used in the validation of Sensys. Regarding postgresql, the version included in the distros is greater, the expectation is to use the 9.3 version which is the one that works with Sensys. Regarding zeromq, it is in the EPEL repo, if it is considered part of the base repos then it can be removed.

koomie commented 7 years ago

Thank you for the submission and apologies in the delay for the review. I regret to inform that the TSC reviewers have recommended to reject this component submission until several changes being requested are completed. The specific requests are highlighted as follows:

harias commented 7 years ago

@koomie thanks for the feedback. We will address it.

ericho commented 7 years ago

Hi @koomie, I just have two questions.

  1. Is the post-install documentation the same that is available here? We want to take an example of format and scope of the needed documentation.
  2. Regarding unit tests, can we take as example the tests available here?

Thank you.

koomie commented 7 years ago

  1. the documentation link you listed is to the common/ area that is shared by all of the current recipes. You will likely want add files in common/ in a fork and reference them in 1 or more recipes. The recipes themselves are currently organized by OS distro/arch/provisioner/resource manager. Here is one example: centos/x86_64/warewulf/slurm. If you having a working latex environment, you can descend into one of these subdirs and run make to generate the pdf doc. You can also run the lexical parser which generates the companion installation script template via: ../../../../parse_doc.pl steps.tex

  2. adding to a subdir of admin/ is probably fine, but you might want to look at other subdirectories if you need examples of tests that run through the resource manager. Note also that the latest release docs have a brief appendix (Appendix C) with a discussion on running the tests. (e.g. here)

harias commented 6 years ago

Hi @koomie,

I want to inform you that Sensys v1.0.0 release was published today, it adds the support of CentOS7.3 and SLES12sp2 to Sensys.

It is addressing the feedback that was provided after the initial integration request.

Among the changes we are including:

We will appreciate any comment you may have. Thank you.

crbaird commented 6 years ago

Hi @marcemq. We would like to pull in your document additions, but we need a slight modification to your 18fc266 commit. The manifests are automatically generated, so if you could make a PR against our obs/OpenHPC_1.3.3_Factory branch containing just the common/sensys.tex and all the steps.tex files, I'll be happy to pull it. Thanks.

marcemq commented 6 years ago

Hi @crbaird. The Sensys post-install configuration without manifests is placed at the branch marcemq/ohpc@483143b which was sent for a pull request based on the obs/OpenHPC_1.3.3_Factory branch.

I'll appreciate any comment that you may have. Thank you.

crbaird commented 6 years ago

Thanks @marcemq.

crbaird commented 6 years ago

We're going to need some changes in order to be able to include sensys in OpenHPC. The orcmd is currently started by hand and forked. This is not standard practice for system daemons, and it causes all sampling messages to appear on the console via stdout. Please add a systemd unit file to control the daemon, and don't write logging events to terminal output. Please also expand the integration tests to show that data is actually being logged as expected using the setup from the installation guide.

harias commented 6 years ago

@crbaird thanks for the feedback. When is this needed in order to proceed with the integration?

koomie commented 6 years ago

It won't be able to make it into the current release as is, so having updates before the end of the year would allow for trying to target the following release (v1.3.4).