getmoto / moto

A library that allows you to easily mock out tests based on AWS infrastructure.
http://docs.getmoto.org/en/latest/
Apache License 2.0
7.62k stars 2.04k forks source link

motoserver doesn't support multi-layer S3 key #5730

Closed dszhengyu closed 1 year ago

dszhengyu commented 1 year ago

I trying the run the following code to check if I can use motoserver on my UT:

import moto
from moto.server import ThreadedMotoServer
import boto3

# start motoserver
server = ThreadedMotoServer()
server.start()

# The model from intro: http://docs.getmoto.org/en/latest/docs/getting_started.html#moto-usage
class MyModel(object):
    def __init__(self, name, value):
        self.name = name
        self.value = value
    def save(self):
        s3 = boto3.client('s3', region_name='us-east-1', endpoint_url="http://localhost:5000")
        s3.put_object(Bucket='mybucket', Key=self.name, Body=self.value)

conn = boto3.resource('s3', region_name='us-east-1', endpoint_url="http://localhost:5000")
conn.create_bucket(Bucket='mybucket')

# This will succeeded
s3_key='steve'
model_instance = MyModel(s3_key, 'is awesome')
model_instance.save()

# This failed
s3_key='steve/shoes'
model_instance = MyModel(s3_key, 'is awesome')
model_instance.save()

I tried to write to two keys, one is 'steve' and another is 'steve/shoes', the first one succeeded and second one failed, command line output:

# python tmp.py
The ThreadedMotoServer is considered in beta for now, and the exact interface and behaviour may still change.
Please let us know if you’d like to see any changes.
========
Starting a new Thread with MotoServer running on 0.0.0.0:5000...
127.0.0.1 - - [02/Dec/2022 18:15:17] "PUT /mybucket HTTP/1.1" 200 -
127.0.0.1 - - [02/Dec/2022 18:15:17] "PUT /mybucket/steve HTTP/1.1" 200 -
127.0.0.1 - - [02/Dec/2022 18:15:17] "PUT /mybucket/steve/aa HTTP/1.1" 404 -
Traceback (most recent call last):
  File "tmp.py", line 24, in <module>
    model_instance.save()
  File "tmp.py", line 14, in save
    s3.put_object(Bucket='mybucket', Key=self.name, Body=self.value)
  File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 395, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/conda/lib/python3.8/site-packages/botocore/client.py", line 725, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the PutObject operation: Not Found

Here is the pip list:

# pip list
Package                     Version
--------------------------- ---------------
absl-py                     1.3.0
aenum                       3.1.11
aiobotocore                 2.3.4
aiohttp                     3.8.1
aioitertools                0.11.0
aiosignal                   1.3.1
antlr4-python3-runtime      4.9.3
apex                        0.1
asn1crypto                  1.5.1
asttokens                   2.0.8
async-timeout               4.0.2
attrs                       21.4.0
aws-sam-translator          1.55.0
aws-xray-sdk                2.11.0
awscli                      1.22.76
awsio                       0.0.1
awswrangler                 2.16.1
backcall                    0.2.0
backoff                     2.2.1
bcrypt                      4.0.0
beautifulsoup4              4.11.1
bokeh                       2.4.3
boto3                       1.21.21
botocore                    1.24.21
brotlipy                    0.7.0
cached-property             1.5.2
cachetools                  5.2.0
cattrs                      22.1.0
certifi                     2022.6.15
cffi                        1.15.1
cfn-lint                    0.72.1
charset-normalizer          2.1.1
click                       8.1.3
cloudpickle                 2.1.0
cmake                       3.18.2.post1
colorama                    0.4.3
coloredlogs                 15.0.1
conda                       4.12.0
conda-content-trust         0.1.3
conda-package-handling      1.8.1
coverage                    6.5.0
cryptography                37.0.4
cycler                      0.11.0
Cython                      0.29.32
decorator                   5.1.1
deepspeed                   0.6.1+0a96707
dgl                         0.9.0
dill                        0.3.5.1
diskcache                   5.4.0
docker                      6.0.1
docutils                    0.15.2
ecdsa                       0.18.0
et-xmlfile                  1.1.0
exceptiongroup              1.0.4
executing                   1.0.0
faiss-gpu                   1.7.2
filelock                    3.8.0
Flask                       2.2.2
Flask-Cors                  3.0.10
flatbuffers                 22.11.23
fonttools                   4.37.1
frozenlist                  1.3.3
fsspec                      2022.7.1
future                      0.18.2
gevent                      21.12.0
gitdb                       4.0.9
GitPython                   3.1.27
google-auth                 2.14.1
google-auth-oauthlib        0.4.6
google-pasta                0.2.0
graphlib-backport           1.0.3
graphql-core                3.2.3
greenlet                    1.1.3
gremlinpython               3.6.1
grpcio                      1.50.0
h5py                        3.7.0
hjson                       3.1.0
horovod                     0.24.3
huggingface-hub             0.11.0
humanfriendly               10.0
hydra-core                  1.2.0
idna                        3.3
imageio                     2.21.2
importlib-metadata          4.12.0
importlib-resources         5.10.0
iniconfig                   1.1.1
inotify-simple              1.2.1
ipython                     8.1.0
isodate                     0.6.1
itsdangerous                2.1.2
jedi                        0.18.1
Jinja2                      3.1.2
jmespath                    1.0.1
joblib                      1.1.0
jschema-to-python           1.2.3
jsii                        1.59.0
jsondiff                    2.0.0
jsonpatch                   1.32
jsonpath-ng                 1.5.3
jsonpickle                  3.0.0
jsonpointer                 2.3
jsonschema                  3.2.0
junit-xml                   1.9
kiwisolver                  1.4.4
libmambapy                  0.22.1
lightning-bolts             0.5.0
llvmlite                    0.36.0
lxml                        4.9.1
MagnusDataloaders           1.0
MagnusFeaturizers           1.0.273
MagnusModelConfigs          1.0
MagnusModelConversion       1.0.272
MagnusModelEvaluation       1.0
MagnusModelInference        1.0.272
MagnusModelTraining         1.0
MagnusModelUtils            1.0
MagnusSageMaker             1.0.247
mamba                       0.22.1
Markdown                    3.4.1
MarkupSafe                  2.1.1
matplotlib                  3.5.3
matplotlib-inline           0.1.6
mmh3                        3.0.0
mock                        4.0.3
more-itertools              9.0.0
moto                        3.1.4
mpi4py                      3.1.3
mpmath                      1.2.1
multidict                   6.0.2
multiprocess                0.70.13
nest-asyncio                1.5.6
networkx                    2.8.6
ninja                       1.10.2.3
numba                       0.53.1
numpy                       1.22.2
oauthlib                    3.2.2
omegaconf                   2.2.3
onnx                        1.12.0
onnxruntime-gpu             1.13.1
opencv-python               4.6.0.66
openpyxl                    3.0.10
opensearch-py               1.1.0
OttoTraining                1.0
OttoTrainingJobLauncher     1.0
packaging                   21.3
pandas                      1.4.4
paramiko                    2.11.0
parso                       0.8.3
pathos                      0.2.9
pbr                         5.11.0
petastorm                   0.12.0
pexpect                     4.8.0
pg8000                      1.29.3
pickleshare                 0.7.5
Pillow                      9.2.0
pip                         22.2.2
plotly                      5.10.0
pluggy                      1.0.0
ply                         3.11
portalocker                 2.5.1
pox                         0.3.1
ppft                        1.7.6.5
progressbar2                4.2.0
prompt-toolkit              3.0.30
protobuf                    3.19.4
protobuf3-to-dict           0.1.5
psutil                      5.9.1
ptyprocess                  0.7.0
pure-eval                   0.2.2
py                          1.11.0
py-cpuinfo                  8.0.0
py4j                        0.10.9.5
pyarrow                     7.0.0
pyasn1                      0.4.8
pyasn1-modules              0.2.8
pybind11                    2.10.0
pycosat                     0.6.3
pycparser                   2.21
pyDeprecate                 0.3.1
pyfunctional                1.4.3
Pygments                    2.13.0
pyinstrument                3.4.2
pyinstrument-cext           0.2.4
PyMySQL                     1.0.2
PyNaCl                      1.5.0
pyOpenSSL                   22.0.0
pyparsing                   3.0.9
pyrsistent                  0.19.2
PySocks                     1.7.1
pyspark                     3.3.1
pytest                      7.1.1
pytest-cov                  3.0.0
pytest-mock                 3.7.0
python-dateutil             2.8.2
python-jose                 3.3.0
python-utils                3.4.5
pytorch-lightning           1.5.10
pytz                        2022.2.1
PyYAML                      5.4.1
pyzmq                       23.2.1
redshift-connector          2.0.909
regex                       2022.10.31
requests                    2.28.1
requests-aws4auth           1.1.2
requests-oauthlib           1.3.1
responses                   0.22.0
retrying                    1.3.3
rsa                         4.7.2
ruamel-yaml-conda           0.15.80
s3fs                        2022.7.1
s3transfer                  0.5.2
sagemaker                   2.102.0
sagemaker-experiments       0.1.35
sagemaker-pytorch-training  2.6.2
sagemaker-training          4.2.6
sarif-om                    1.0.4
scikit-learn                1.1.2
scipy                       1.9.1
scramp                      1.4.4
seaborn                     0.11.2
sentencepiece               0.1.97
setuptools                  59.5.0
shap                        0.41.0
six                         1.16.0
sklearn                     0.0
slicer                      0.0.7
smclarify                   0.2
smdebug                     1.0.19b20220829
smdebug-rulesconfig         1.0.1
smdistributed-dataparallel  1.5.1
smdistributed-modelparallel 1.11.0
smmap                       5.0.0
soupsieve                   2.3.2.post1
sshpubkeys                  3.3.1
stack-data                  0.5.0
sympy                       1.11.1
tabulate                    0.8.10
tenacity                    8.0.1
tensorboard                 2.11.0
tensorboard-data-server     0.6.1
tensorboard-plugin-wit      1.8.1
threadpoolctl               3.1.0
tokenizers                  0.13.2
toml                        0.10.2
tomli                       2.0.1
torch                       1.12.0+cu113
torchaudio                  0.12.0+cu113
torchdata                   0.4.0
torchfile                   0.1.0
torchmetrics                0.10.3
torchnet                    0.0.4
torchvision                 0.13.0+cu113
tornado                     6.2
tqdm                        4.64.0
traitlets                   5.3.0
transformers                4.24.0
types-toml                  0.10.8.1
typing_extensions           4.3.0
urllib3                     1.26.11
visdom                      0.1.8.9
wcwidth                     0.2.5
websocket-client            1.4.0
Werkzeug                    2.2.2
wheel                       0.37.1
wrapt                       1.14.1
xmltodict                   0.13.0
yarl                        1.8.1
zipp                        3.8.1
zope.event                  4.5.0
zope.interface              5.4.0

Please check if there is anything I need to supplement for the debugging, thx.

bblommers commented 1 year ago

Hi @dszhengyu, earlier Moto versions do not play well with werkzeug 2.2.2. If you upgrade Moto to >= 4.0.8 (or downgrade werkzeug to 2.1.2), it should work just fine.

(Related: #5341 )

dszhengyu commented 1 year ago

Thx about the hint! It works now 😁

bblommers commented 1 year ago

Happy to help @dszhengyu! I'll close this, but let us know if you run into any other issues.