search

gridengine / config-api

Univa Grid Engine Python Configuration API
http://www.univa.com
Apache License 2.0
7 stars 3 forks source link

UGE project addition is not working #3

Open kmr210 opened 4 years ago

kmr210 commented 4 years ago

HI Team,

When i try to add a project, always throws an error saying ObjectAlreadyExists.

Please help me to resolve this issue,

Below is the problem snippet,

n [19]: print(pp.to_json())
{"object_version": "1.0", "object_class": "Project", "modified_by": "kmr", "modified_on": "2020-07-31T09:35:54.129700", "data": {"name": "test123", "oticket": 0, "fshare": 100, "acl": null, "xacl": null}}

In [30]: pp2=api.add_prj(pycl_object=pp,name=None, data=None, metadata=None, jso ...: n_string=None)

ObjectAlreadyExists Traceback (most recent call last)

in ----> 1 pp2=api.add_prj(pycl_object=pp,name=None, data=None, metadata=None, json_string=None) in add_prj(self, pycl_object, name, data, metadata, json_string) qconf_api.py in wrapped_call(func, *args, **kwargs) 155 def wrapped_call(func, *args, **kwargs): 156 try: --> 157 result = func(*args, **kwargs) 158 return result 159 except QconfException as ex: qconf_api.py in add_prj(self, pycl_object, name, data, metadata, json_string) 1470 return self.project_manager.add_object( 1471 pycl_object=pycl_object, name=name, data=data, -> 1472 metadata=metadata, json_string=json_string) 1473 1474 @api_call dict_based_object_manager.py in add_object(self, pycl_object, name, data, metadata, json_string) 117 try: 118 old_object = self.get_object(object_name) --> 119 raise ObjectAlreadyExists('%s %s already exists.' % (self.OBJECT_CLASS_NAME, object_name)) 120 except ObjectNotFound as ex: 121 # ok ObjectAlreadyExists: Project test123 already exists.
kmr210 commented 4 years ago

any update on this issue is greatly appreciated.

Thanks

alea55 commented 4 years ago

Hi,

did you verify that the prerequisite is given, i.e. the project does not already exist ? On the commandline you can verify with: source your UGE env qconf -sprjl

you should not see test123

you can this do programmatically as well by see the config-api/test/test_project.py example...

Andre

kmr210 commented 4 years ago

Andre,

Thanks for the prompt reply!

I verified it doesn't exist. I followed exact same steps given in the testsuite.

BTW, i have tried this both in Python2.7 and 3.6 versions.

Thanks

alea55 commented 4 years ago

So did you try this one and isn't this working for you ?

cd config-api nosetests -s test/test_project.py

this works ok for me (8.6.x = V86_BRANCH and 8.7.x master)

kmr210 commented 4 years ago

HI,

I have UGE 8.6.7 version deployed, i tried using python2.7/3.6 versions to run PyCl, both failed.

I have updated 'test_values.json' with 2 test projects, you can see it complains 'ObjectAlreadyExists' error. I made sure those 2 test projects are unique in GRD.

Even ACL/Queue, addition also complains the same.

Below is the error log,

F.E..E.EF....

ERROR: test.test_project.test_add_prj

Traceback (most recent call last): File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(self.arg) File "/demo/config-api/test/test_project.py", line 64, in test_add_prj project = API.add_prj(name=PROJECT_NAME) File "", line 2, in add_prj File "/demo/config-api/uge/api/qconf_api.py", line 157, in wrapped_call result = func(args, **kwargs) File "/demo/config-api/uge/api/qconf_api.py", line 1472, in add_prj metadata=metadata, json_string=json_string) File "/demo/config-api/uge/api/impl/dict_based_object_manager.py", line 119, in add_object raise ObjectAlreadyExists('%s %s already exists.' % (self.OBJECT_CLASS_NAME, object_name)) uge.exceptions.object_already_exists.ObjectAlreadyExists: Project 3p4N2H already exists. -------------------- >> begin captured logging << -------------------- UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl] UgeSubprocess: DEBUG: Exit status: 0 UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H] UgeSubprocess: DEBUG: Exit status: 0 --------------------- >> end captured logging << ---------------------

====================================================================== ERROR: test.test_project.test_get_prj

Traceback (most recent call last): File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/demo/config-api/test/test_project.py", line 87, in test_get_prj assert (project.data['name'] == PROJECT_NAME) KeyError: 'name' -------------------- >> begin captured logging << -------------------- UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H] UgeSubprocess: DEBUG: Exit status: 0 --------------------- >> end captured logging << ---------------------

====================================================================== ERROR: test.test_project.test_modify_prj

Traceback (most recent call last): File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/demo/config-api/test/test_project.py", line 111, in test_modify_prj oticket = project.data['oticket'] KeyError: 'oticket' -------------------- >> begin captured logging << -------------------- UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H] UgeSubprocess: DEBUG: Exit status: 0 --------------------- >> end captured logging << ---------------------

====================================================================== FAIL: test.test_project.test_object_not_found

Traceback (most recent call last): File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(self.arg) File "/demo/config-api/test/utils.py", line 169, in inner return func(args, **kwargs) File "/demo/config-api/test/test_project.py", line 47, in test_object_not_found assert (False) AssertionError: <uge.objects.project_v1_0.Project object at 0x7fffe9107cf8> = <uge.api.qconf_api.QconfApi object at 0x7fffea2246d8>.get_prj('non_existent_project')

assert (False)

-------------------- >> begin captured logging << -------------------- QconfApi: DEBUG: Configuration: SGE_ROOT=/default, SGE_CELL=tform, SGE_QMASTER_PORT=6444, SGE_EXECD_PORT=6445 QconfExecutor: TRACE: Retrieving UGE version UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -help] UgeSubprocess: DEBUG: Exit status: 0 QconfExecutor: DEBUG: UGE version: 8.6.7 UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj non_existent_project] UgeSubprocess: DEBUG: Exit status: 0 --------------------- >> end captured logging << ---------------------

====================================================================== FAIL: test.test_project.test_delete_prj

Traceback (most recent call last): File "/venv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest self.test(*self.arg) File "/demo/config-api/test/test_project.py", line 125, in test_delete_prj assert (len(project_list2) == len(project_list) - 1) AssertionError:

-------------------- >> begin captured logging << -------------------- UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl] UgeSubprocess: DEBUG: Exit status: 0 UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprj 3p4N2H] UgeSubprocess: DEBUG: Exit status: 0 UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -dprj 3p4N2H] UgeSubprocess: DEBUG: Exit status: 0 UgeSubprocess: DEBUG: from [qconf_executor.py:78] Invoking: [. /settings.sh; qconf -sprjl] UgeSubprocess: DEBUG: Exit status: 0 --------------------- >> end captured logging << ---------------------

Name Stmts Miss Cover

uge/init.py 1 0 100% uge/api/init.py 2 0 100% uge/api/ar_api.py 95 59 38% uge/api/impl/init.py 0 0 100% uge/api/impl/access_list_manager.py 49 26 47% uge/api/impl/admin_host_manager.py 13 0 100% uge/api/impl/ar_manager.py 16 1 94% uge/api/impl/calendar_manager.py 16 0 100% uge/api/impl/checkpointing_environment_manager.py 16 0 100% uge/api/impl/cluster_configuration_manager.py 34 15 56% uge/api/impl/cluster_queue_manager.py 16 0 100% uge/api/impl/complex_configuration_manager.py 45 21 53% uge/api/impl/dict_based_object_manager.py 191 43 77% uge/api/impl/dict_list_based_object_manager.py 82 51 38% uge/api/impl/execution_host_manager.py 30 8 73% uge/api/impl/host_group_manager.py 17 0 100% uge/api/impl/job_class_manager.py 16 0 100% uge/api/impl/list_based_object_manager.py 57 34 40% uge/api/impl/manager_manager.py 13 0 100% uge/api/impl/operator_manager.py 13 0 100% uge/api/impl/parallel_environment_manager.py 16 0 100% uge/api/impl/project_manager.py 19 0 100% uge/api/impl/qconf_executor.py 101 55 46% uge/api/impl/qrdel_executor.py 71 50 30% uge/api/impl/qrstat_executor.py 93 68 27% uge/api/impl/qrsub_executor.py 74 53 28% uge/api/impl/resource_quota_set_manager.py 16 0 100% uge/api/impl/scheduler_configuration_manager.py 21 3 86% uge/api/impl/share_tree_manager.py 50 26 48% uge/api/impl/submit_host_manager.py 13 0 100% uge/api/impl/user_manager.py 16 0 100% uge/api/qconf_api.py 408 135 67% uge/config/init.py 0 0 100% uge/config/config_manager.py 152 29 81% uge/constants/init.py 0 0 100% uge/constants/uge_status.py 10 0 100% uge/exceptions/init.py 10 0 100% uge/exceptions/ar_exception.py 32 18 44% uge/exceptions/authorization_error.py 5 1 80% uge/exceptions/command_failed.py 14 7 50% uge/exceptions/configuration_error.py 5 1 80% uge/exceptions/invalid_argument.py 5 1 80% uge/exceptions/invalid_request.py 5 1 80% uge/exceptions/object_already_exists.py 5 0 100% uge/exceptions/object_not_found.py 5 1 80% uge/exceptions/qconf_exception.py 32 11 66% uge/exceptions/qmaster_unreachable.py 5 1 80% uge/log/init.py 0 0 100% uge/log/log_manager.py 143 53 63% uge/log/logger_factory.py 38 3 92% uge/log/stream_log_handler.py 12 3 75% uge/log/timed_rotating_file_log_handler.py 12 3 75% uge/log/trace_log_record.py 13 0 100% uge/log/trace_logger.py 22 5 77% uge/objects/init.py 0 0 100% uge/objects/ar_object_factory.py 68 51 25% uge/objects/project_v1_0.py 12 0 100% uge/objects/qconf_name_list.py 23 9 61% uge/objects/qconf_object.py 251 105 58% uge/objects/qconf_object_factory.py 154 78 49% uge/objects/uge_release_object_map.py 49 0 100% uge/utility/init.py 0 0 100% uge/utility/uge_subprocess.py 75 38 49%

TOTAL 2777 1067 62%

Ran 13 tests in 6.010s

FAILED (errors=3, failures=2)

alea55 commented 4 years ago

Maybe there is a mismatch between the current config-api implementation and UGE 8.6.7. Is it possible for you to upgrade your cluster to a more recent UGE patch version ? We arrived already at 8.6.14 and there have been many issues fixed.

alea55 commented 4 years ago

I installed the 8.6.7 bits and proceeded as follows:

  1. install all packages (including the config-api package)
  2. install UGE as usual
  3. add additional python modules (python-nose*, python-decorator, python-xmltodict, not sure if python3 was already completely working for 8.6.7)
  4. cd $SGE_ROOT/config-api, tar xvzf tar xvzf uge-pycl-8.6.7.tar.gz, cd uge-pycl-8.6.7/
  5. there are two files missing test/utils.py and test_values.json copy them in from the opensource repo
  6. start your cluster and source the UGE env
  7. nosetest -s test/test_project.py runs successful then for me

There seems to be a problem with the packaging of the test dir content since these two files should of course be present...

kmr210 commented 4 years ago

Thanks for your valuable inputs.

We can't upgrade UGE to latest version at the moment as lot of co-ordination required.

In my case, test dir has all the files required, but it is not letting project/queue addition, not sure why.

How about UGE Cli, can you please through some light on how to use it? so that for missing portions, i will run 'qconf' commands directly.

Thanks a lot!

alea55 commented 4 years ago

see AdminsGuideGE.pdf, section 1.6.6 (or search for projects)

kmr210 commented 4 years ago

HI,

Assuming, you are referring to 'adminsguide' related to UGE PycL, where to find it? please let me know

I could see only developer_guide.md in this repo.

Thanks

alea55 commented 4 years ago

It is part of UGE gridengine packages. What do you get if you do: qstat -help | head -1 Usually you find the pdf documentation under $SGE_ROOT/doc. If it is not there you have to install your version specific doc package of UGE. I assume that you are running UGE and no other gridengine distribution or am I wrong ?