sassoftware / python-sasctl

Python package and CLI for user-friendly integration with SAS Viya
https://sassoftware.github.io/python-sasctl
Apache License 2.0
46 stars 41 forks source link

SASCTL Register_model fails to register a second model in an existing project #162

Closed EdwinvanUnen closed 1 year ago

EdwinvanUnen commented 1 year ago

Describe the issue When registering a (first) model into a MM-project it creates the project in case it doesn't exist. But when I try to register a second model it fails because "the project is already in use". Note: the second model is succesfully put into the Public repository with new version, but not in the MM project.

To Reproduce This is the code: first creating the astore, then registering the model:

df.regression.logistic(target='Species', inputs=features, savestate={"caslib":'Casuser',"name":'iris_model_table',"Replace":'True'}) astore = sess.CASTable('iris_model_table') with Session(host, username='snlevu',password=password): model = register_model(astore, 'iris logistic3', 'EVU_Iris3', force=True)

Full Error:

HTTPError Traceback (most recent call last) /tmp/ipykernel_32401/2443453042.py in 1 with Session(host, username='snlevu',password=password): ----> 2 model = register_model(astore, 'iris logistic3', 'EVU_Iris3', force=True)

/opt/anaconda3/lib/python3.8/site-packages/sasctl/tasks.py in register_model(model, name, project, repository, input, version, files, force, record_packages) 299 model_props = {} 300 --> 301 project = _create_project(project, model_props, repo_obj, 302 invar, outvar) 303

/opt/anaconda3/lib/python3.8/site-packages/sasctl/tasks.py in _create_project(project_name, model, repo, input_vars, output_vars) 147 properties['targetLevel'] = None 148 --> 149 project = mr.create_project(project_name, repo, 150 variables=input_vars + output_vars, 151 **properties)

/opt/anaconda3/lib/python3.8/site-packages/sasctl/_services/model_repository.py in create_project(cls, project, repository, **kwargs) 436 437 project.update(kwargs) --> 438 return cls.post('/projects', json=project, 439 headers={ 440 'Content-Type':

/opt/anaconda3/lib/python3.8/site-packages/sasctl/_services/service.py in post(cls, *args, kwargs) 115 def post(cls, *args, *kwargs): 116 """Send a POST request.""" --> 117 return cls.request('post', args, kwargs) 118 119 @classmethod

/opt/anaconda3/lib/python3.8/site-packages/sasctl/_services/service.py in request(cls, verb, path, session, raw, format, kwargs) 95 path = cls._SERVICE_ROOT + '/' + path 96 ---> 97 return core.request(verb, path, session, raw, format, kwargs) 98 99 @classmethod

/opt/anaconda3/lib/python3.8/site-packages/sasctl/core.py in request(verb, path, session, raw, format, **kwargs) 1138 1139 if 400 <= response.status_code <= 599: -> 1140 raise HTTPError(response.url, response.status_code, response.text, 1141 response.headers, None) 1142

HTTPError: HTTP Error 409: {"version":2,"httpStatusCode":409,"errorCode":21201,"message":"The name \"EVU_Iris3\" is already in use within the selected folder. Enter a unique name.","details":["path: /modelRepository/projects","correlator: f5d79f76-60e3-44e1-83a6-b0a8360710b3"]}

Expected behavior I expect the system to add the second model in the Model Manager project EVU_Iris3. But it fails because it already exists. It puts the model correctly in the Public Repository with a new version nr.

** Viya-Server: https://viya401.zeus.sashq-d.openstack.sas.com/

Version SASCTL version: 1.5.4

smlindauer commented 1 year ago

I recreated this issue in v1.5.4 and found that the bug did not persist in the more recent releases of sasctl.

Upon further investigation, I discovered that this is a known issue from 1.5.4 caused by the usage of Viya 3.4's filter method (specifically using modelRepository/projects/?properties=(name,%20EVU_Iris3) in this example instead of modelRepository/projects?filter=eq(name,%20EVU_Iris3)). This was fixed in 9df0eee76456d3d35c4d7b414a833293e6da8ac9, which maps to sasctl version 1.7.3.

Upgrading to v1.7.3 will fix the error, however I would suggest upgrading to the most recent release of v1.9.1.

EdwinvanUnen commented 1 year ago

We did an upgrade to v1.9.1, but it still fails:

urllib.error.HTTPError: HTTP Error 409: {"version":2,"httpStatusCode":409,"errorCode":21201,"message":"The name \"EVU_Iris6\" is already in use within the selected folder. Enter a unique name.","details":["path: /modelRepository/projects","correlator: 4601a2c4-115f-462a-bcec-bf53a659ff83"]}

The first registration creates the project with an empty Version 1 and a Version 2 with the first model in it. The second registration fails, also when setting the version nr.

From: Scott Lindauer @.> Sent: donderdag 4 mei 2023 16:16 To: sassoftware/python-sasctl @.> Cc: Edwin van Unen @.>; Author @.> Subject: Re: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

EXTERNAL

I recreated this issue in v1.5.4 and found that the bug did not persist in the more recent releases of sasctl.

Upon further investigation, I discovered that this is a known issue from 1.5.4 caused by the usage of Viya 3.4's filter method (specifically using modelRepository/projects/?properties=(name,%20EVU_Iris3) in this example instead of modelRepository/projects?filter=eq(name,%20EVU_Iris3)). This was fixed in 9df0eeehttps://github.com/sassoftware/python-sasctl/commit/9df0eee76456d3d35c4d7b414a833293e6da8ac9, which maps to sasctl version 1.7.3https://github.com/sassoftware/python-sasctl/releases/tag/v1.7.3.

Upgrading to v1.7.3 will fix the error, however I would suggest upgrading to the most recent release of v1.9.1.

- Reply to this email directly, view it on GitHubhttps://github.com/sassoftware/python-sasctl/issues/162#issuecomment-1534862700, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJSNUMAWCAJ5ANYEP23UAPLXEO2ZFANCNFSM6AAAAAAXVZF5XY. You are receiving this because you authored the thread.Message ID: @.***>

smlindauer commented 1 year ago

Can you add the Session logger and rerun the call?

with Session(host, username='snlevu',password=password) as sasctl_sess: 
    sasctl_sess.add_stderr_logger(level=20)
    model = register_model(astore, 'iris logisticX', 'EVU_IrisX', force=True)

When I run the call, this is the output I get from the debug logger:

>>> model = register_model(astore, "Model Test Test2", "EVU_Iris3", force=True) 

HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 GET /modelRepository/repositories HTTP/1.1 200 /modelRepository/repositories HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 POST /modelRepository/models HTTP/1.1 201 /modelRepository/models

I have scrubbed the server name and bold the parts that should be fixed for the error. If you are still seeing ?properties= instead of ?filter=eq, then I suspect that the sasctl upgrade did not work. If so, please verify in your sasctl version by running:

python -c "import sasctl; print(sasctl.__version__)"

If it is still the older version, rerun the pip install command with the --upgrade option:

pip install -U sasctl
EdwinvanUnen commented 1 year ago

Ah, yes, I was still seeing ?properties=

HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories

And version 1.5.4

But after running pip command it remains 1.5.4:

@.*** Adding Wouter to the email, he maintains the env.

From: Scott Lindauer @.> Sent: donderdag 4 mei 2023 18:39 To: sassoftware/python-sasctl @.> Cc: Edwin van Unen @.>; Author @.> Subject: Re: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

You don't often get email from @.*** Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

EXTERNAL

Can you add the Session logger and rerun the call?

with Session(host, username='snlevu',password=password) as sasctl_sess:

sasctl_sess.add_stderr_logger(level=20)

model = register_model(astore, 'iris logisticX', 'EVU_IrisX', force=True)

When I run the call, this is the output I get from the debug logger:

model = register_model(astore, "Model Test Test2", "EVU_Iris3", force=True)

HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 GET /modelRepository/repositories HTTP/1.1 200 /modelRepository/repositories HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 POST /modelRepository/models HTTP/1.1 201 /modelRepository/models

I have scrubbed the server name and bold the parts that should be fixed for the error. If you are still seeing ?properties= instead of ?filter=eq, then I suspect that the sasctl upgrade did not work. If so, please verify in your sasctl version by running:

python -c "import sasctl; print(sasctl.version)"

If it is still the older version, rerun the pip install command with the --upgrade option:

pip install -U sasctl

- Reply to this email directly, view it on GitHubhttps://github.com/sassoftware/python-sasctl/issues/162#issuecomment-1535058481, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJSNUMFRHT7OVSHZHPURQ43XEPLRFANCNFSM6AAAAAAXVZF5XY. You are receiving this because you authored the thread.Message ID: @.***>

EdwinvanUnen commented 1 year ago

Hey Edwin,

You must have used a locally installed version of sasctl, because when I run this in a brand new python notebook, I get the correct version : @.***

And why would you use a "pip install" when sasctl is already installed ?

Bye, Wouter. From: Edwin van Unen @.> Sent: Thursday, 4 May 2023 19:17 To: sassoftware/python-sasctl @.>; sassoftware/python-sasctl @.> Cc: Wouter Van de Weghe @.> Subject: RE: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

Ah, yes, I was still seeing ?properties=

HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories

And version 1.5.4

But after running pip command it remains 1.5.4:

@.*** Adding Wouter to the email, he maintains the env.

From: Scott Lindauer @.**@.>> Sent: donderdag 4 mei 2023 18:39 To: sassoftware/python-sasctl @.**@.>> Cc: Edwin van Unen @.**@.>>; Author @.**@.>> Subject: Re: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

EXTERNAL

Can you add the Session logger and rerun the call?

with Session(host, username='snlevu',password=password) as sasctl_sess:

sasctl_sess.add_stderr_logger(level=20)

model = register_model(astore, 'iris logisticX', 'EVU_IrisX', force=True)

When I run the call, this is the output I get from the debug logger:

model = register_model(astore, "Model Test Test2", "EVU_Iris3", force=True)

HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 GET /modelRepository/repositories HTTP/1.1 200 /modelRepository/repositories HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 POST /modelRepository/models HTTP/1.1 201 /modelRepository/models

I have scrubbed the server name and bold the parts that should be fixed for the error. If you are still seeing ?properties= instead of ?filter=eq, then I suspect that the sasctl upgrade did not work. If so, please verify in your sasctl version by running:

python -c "import sasctl; print(sasctl.version)"

If it is still the older version, rerun the pip install command with the --upgrade option:

pip install -U sasctl

- Reply to this email directly, view it on GitHubhttps://github.com/sassoftware/python-sasctl/issues/162#issuecomment-1535058481, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJSNUMFRHT7OVSHZHPURQ43XEPLRFANCNFSM6AAAAAAXVZF5XY. You are receiving this because you authored the thread.Message ID: @.**@.>>

EdwinvanUnen commented 1 year ago

Hi,

Issue solved with help of Wouter.

Cheers, Edwin

From: Wouter Van de Weghe @.> Sent: vrijdag 5 mei 2023 09:03 To: Edwin van Unen @.>; sassoftware/python-sasctl @.>; sassoftware/python-sasctl @.> Subject: RE: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

Hey Edwin,

You must have used a locally installed version of sasctl, because when I run this in a brand new python notebook, I get the correct version : @.***

And why would you use a "pip install" when sasctl is already installed ?

Bye, Wouter. From: Edwin van Unen @.**@.>> Sent: Thursday, 4 May 2023 19:17 To: sassoftware/python-sasctl @.**@.>>; sassoftware/python-sasctl @.**@.>> Cc: Wouter Van de Weghe @.**@.>> Subject: RE: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

Ah, yes, I was still seeing ?properties=

HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/projects?properties=(name,%20EVU_IrisX) HTTP/1.1 GET https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories HTTP/1.1 200 https://viya401.zeus.sashq-d.openstack.sas.com/modelRepository/repositories

And version 1.5.4

But after running pip command it remains 1.5.4:

@.*** Adding Wouter to the email, he maintains the env.

From: Scott Lindauer @.**@.>> Sent: donderdag 4 mei 2023 18:39 To: sassoftware/python-sasctl @.**@.>> Cc: Edwin van Unen @.**@.>>; Author @.**@.>> Subject: Re: [sassoftware/python-sasctl] SASCTL Register_model fails to register a second model in an existing project (Issue #162)

You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification

EXTERNAL

Can you add the Session logger and rerun the call?

with Session(host, username='snlevu',password=password) as sasctl_sess:

sasctl_sess.add_stderr_logger(level=20)

model = register_model(astore, 'iris logisticX', 'EVU_IrisX', force=True)

When I run the call, this is the output I get from the debug logger:

model = register_model(astore, "Model Test Test2", "EVU_Iris3", force=True)

HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 GET /modelRepository/repositories HTTP/1.1 200 /modelRepository/repositories HTTP/1.1 GET /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 200 /modelRepository/projects?filter=eq(name,%20%22EVU_Iris3%22) HTTP/1.1 GET /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 200 /modelRepository/projects/d830b733-80f7-448f-8e08-f216798fac95 HTTP/1.1 POST /modelRepository/models HTTP/1.1 201 /modelRepository/models

I have scrubbed the server name and bold the parts that should be fixed for the error. If you are still seeing ?properties= instead of ?filter=eq, then I suspect that the sasctl upgrade did not work. If so, please verify in your sasctl version by running:

python -c "import sasctl; print(sasctl.version)"

If it is still the older version, rerun the pip install command with the --upgrade option:

pip install -U sasctl

- Reply to this email directly, view it on GitHubhttps://github.com/sassoftware/python-sasctl/issues/162#issuecomment-1535058481, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJSNUMFRHT7OVSHZHPURQ43XEPLRFANCNFSM6AAAAAAXVZF5XY. You are receiving this because you authored the thread.Message ID: @.**@.>>