Scikit processing example - Client LogStream error

[gerold-csendes-epam]

When running the the local scikit-learn processing srcipt on a notebook instance the training job finishes successfully but I get the following error (I substituted the user and and resource id-s with ....): ClientError: An error occurred (AccessDeniedException) when calling the DescribeLogStreams operation: User: ... is not authorized to perform: logs:DescribeLogStreams on resource: ... -group:/aws/sagemaker/ProcessingJobs:log-stream

It seems like it is trying to write logs which is not really necessary in the local mode, is it? Maybe the issue is that I run the script on sagemaker?

[eitansela]

Hello @gerold-csendes-epam It could be a bug in SageMaker SDK. Can you please attach the logs and the output of pip freeze command? Thanks

[gerold-csendes-epam]

What do you exactly mean by logs? Do you mean Cloudwatch logs? I checked and there is none written when submitting the local processing job from the tutorial notebook.

Here is the result of the pip freeze:

aiobotocore @ file:///tmp/build/80754af9/aiobotocore_1615487469391/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1614280892404/work
aioitertools @ file:///home/conda/feedstock_root/build_artifacts/aioitertools_1605140085294/work
async-timeout==3.0.1
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1605083924122/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache==1.6.1
bcrypt==3.2.0
boto3 @ file:///home/conda/feedstock_root/build_artifacts/boto3_1616540878956/work
botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1616536672126/work
brotlipy==0.7.0
category-encoders @ file:///home/conda/feedstock_root/build_artifacts/category_encoders_1588199510095/work/dist
certifi==2020.12.5
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1606236101993/work
chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1610093490430/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1615405999564/work
decorator==4.4.2
distro==1.5.0
docker==4.4.4
docker-compose==1.28.6
dockerpty==0.4.1
docopt==0.6.2
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1614107550004/work
google-pasta==0.2.0
idna==2.10
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1615778734765/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1613852938555/work/dist/ipykernel-5.5.0-py3-none-any.whl
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1614393573755/work
ipython-genutils==0.2.0
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1610146791023/work
jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1589369830981/work
joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1612898609989/work
jsonschema==3.2.0
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1615693636836/work
jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1612125275706/work
mock @ file:///home/conda/feedstock_root/build_artifacts/mock_1610094566888/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1610319019165/work
numexpr @ file:///home/conda/feedstock_root/build_artifacts/numexpr_1614971340455/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1612953005674/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1612459636436/work
pandas==1.2.3
paramiko==2.7.2
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1607618318316/work
patsy==0.5.1
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1616432837031/work
protobuf==3.15.6
protobuf3-to-dict @ file:///home/conda/feedstock_root/build_artifacts/protobuf3-to-dict_1610115095957/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1615243893546/work
PyNaCl==1.4.0
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyrsistent==0.17.3
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1610291447907/work
python-dateutil==2.8.1
python-dotenv==0.15.0
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1612179539967/work
PyYAML==5.4.1
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1614611703677/work
requests==2.25.1
s3fs @ file:///home/conda/feedstock_root/build_artifacts/s3fs_1608058532532/work
s3transfer @ file:///home/conda/feedstock_root/build_artifacts/s3transfer_1616166519874/work
sagemaker==2.31.1
scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1605985735074/work
scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1615955928702/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
smdebug-rulesconfig @ file:///home/conda/feedstock_root/build_artifacts/smdebug-rulesconfig_1608558075041/work
statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1612273599609/work
tables @ file:///home/conda/feedstock_root/build_artifacts/pytables_1610156075390/work
texttable==1.6.3
threadpoolctl @ file:///tmp/tmp79xdzxkt/threadpoolctl-2.1.0-py3-none-any.whl
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1610094706440/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work
typing-extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1602702424206/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1615828766818/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
websocket-client==0.58.0
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1610094884173/work
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1610354134018/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1614945704755/work

[eitansela]

Got it. I see you are using the latest version of SageMaker SDK. Logs I mean by the output of the Pytjon program. Just to make sure again, the processing job finishes successfully, but you get the error message about the describe logs IAM role?

[gerold-csendes-epam]

The output I get is the one that I shared in the description. The message just gets truncated for some reason. This is the last thing that gets outputted before the error:

405xfy7qlb-algo-1-xyh7a | Processing Complete
405xfy7qlb-algo-1-xyh7a exited with code 0
Aborting on container exit...
===== Job Complete =====

Yeah, your description is correct. Additionally, I can get rid of the error by specifying logs=False argument.

[eitansela]

OK. You can open an issue in SageMaker Python SDK GitHub Repo, and it will be handled as per Service Team priority.