joachimweyl
commented
1 year ago @rob-baron would this benefit from review or is this just for your use for now?
Closed rob-baron closed 1 year ago
joachimweyl
commented
1 year ago @rob-baron would this benefit from review or is this just for your use for now?
rob-baron
commented
1 year ago @joachimweyl yes, review away.
larsks
commented
1 year ago I made some comments yesterday about storage abstractions and ORM, and I thought I should back those up with some examples. I spent some time this morning putting together this repository to demonstrate a few of the points I mentioned in my review:
The repository implements an abstract storage interface with multiple backends, including filesystem and database support.
The database backend uses sqlalchemy, which allows it to operate against sqlite/postgres/mariadb/etc without any code changes.
The code is thoroughly tested.
rob-baron
commented
1 year ago @larsks
Thanks.
rob-baron
commented
1 year ago @larsks,
It really depends on the larger design choices that we make regarding future direction of this project. Sure, I could write in a more object oriented style, but for most database work and much of what has been done here, I found a more procedural to be simpler to write and figure out what the requirements are. I have been more focused on getting this up and running than on what the ultimate design should be.
1) Are we going to target a larger audience for this, or will this remain a utility that we deploy and use? 2) What is the set of services that we expect to require to deploy projects? 3) Are we going to just target OpenShift, or are we also going to support a more general form of kubernetes? 4) How prescriptive are we going to be?
I have been cutting back on how flexible the code that I write is as I am constantly reminded that nobody is asking for that flexibility, and we need to get this done soon! Personally, I have been more concerned with the ability to add new openstack clusters without changing any code, and making used of the Kubernetes scheduler to schedule containers on nodes that are not busy. It is this latter design choice that let me to write a quick utility to share configuration files using the database - as there wasn't any other way I had at that time.
From my perspective, where we provide flexibility really depends on how we answer broader questions.
disclaimer, I will probably edit this in the future.
larsks
commented
1 year ago It really depends on the larger design choices that we make regarding future direction of this project.
I think that as you write tests for this code you may end up seeing the advantage of some of the changes I've suggested; I believe the code -- not just the "share files via the database" part, but also the other code in this PR -- is going to be difficult to test in its current form and will need to be restructured.
larsks
commented
1 year ago These tests appear to have some external dependencies:
$ pytest test
.
.
.
E mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused)
../../../.local/share/virtualenvs/xdmod-cntr-Sc9gjwFV/lib64/python3.11/site-packages/mysql/connector/network.py:511: InterfaceError
=========================================================== short test summary info ===========================================================
ERROR test/test_001.py::test_empty_dataset - mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused)
ERROR test/test_001.py::test_2PIs_3Projects - mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused)
ERROR test/test_001.py::test_removal_project_and_pi - mysql.connector.errors.InterfaceError: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused)
============================================================== 3 errors in 0.20s ==============================================================If the tests require a database server, they should take care of setting one up and tearing it down (or they should mock out the database interactions, or use a sqllite database for testing, etc).
rob-baron
commented
1 year ago Ok, all tests are now in a docker container, instructions to run them are in the README.
% docker run -u root --entrypoint /bin/bash -it test
/app# /usr/bin/mysqld_safe --basedir=/usr --init-file=/app/test_mariadb_init.sql &
/app# /usr/bin/mysqld_safe: Deprecated program name. It will be removed in a future release, use 'mariadb-safe' instead
230719 21:40:31 mysqld_safe Logging to syslog.
230719 21:40:31 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
root@770a19f8afd7:/app# pytest
========================= test session starts =========================
platform linux -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /app
plugins: mock-3.11.1
collected 3 items
test/test_001.py ... [100%]
========================= 3 passed in 0.10s =========================
/app# Note: did do some editing to shorten some of the lines.
This is no longer in draft.
This combines data from coldfront and from the metadata that is stored in our instance of keycloak to produce the set of hierarchy files that are used by xdmod.
The hierarchy as we have implemented is as follows:
Xdmod has a fixed 3 level hierarchy, with an implicit 4th layer that allows to map the job/cloud data to the bottom layer of their 3 level hierarchy (in our case the bottom layer is the PI). Unfortunately, xdmod considers this 4 layer the "group" or the "PI" name as that is how they keep track of jobs as there are no project names on the HPC side.
issues:
192
174
160
159
155