Initial work to automate running load tests against a deployed simulator instance
Load tests
Load test: no limits and no added latency - to test baseline performance
Load test: no limits and 1s latency - to test simulated latency is correct
Load test: 10 requests per second limit - to test that the RPS limiting is correct
Load test: 100,000 tokens per minute limit - to test that the TPM limiting is correct
Improve lorem ipsum text generation performance
use cached token strings and assemble response content from these to avoid repeated calls to lorem generation token counting
Fix: include streaming responses in token counts
Update metric names for token metrics to match latency naming
Split out prompt vs completion tokens using metric dimensions
Replace python-lorem with custom implementation to resolve build errors (see below)
Add docs page for simulator metrics
Closes #6
Build output for failed python-lorem installation [72 lines of output]
/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/dist.py:478: SetuptoolsDeprecationWarning: Invalid dash-separated options
!!
********************************************************************************
Usage of dash-separated 'description-file' will not be supported in future
versions. Please use the underscore name 'description_file' instead.
By 2024-Sep-26, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
!!
opt = self.warn_dash_deprecation(opt, section)
running dist_info
creating /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info
writing /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/dependency_links.txt
writing requirements to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/requires.txt
writing top-level names to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '.DS_Store' found anywhere in distribution
warning: no files found matching '*.rst' anywhere in distribution
no previously-included directories found matching '.eggs'
no previously-included directories found matching '.github'
no previously-included directories found matching '.venv'
no previously-included directories found matching 'docs'
no previously-included directories found matching 'temp'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt'
creating '/tmp/pip-modern-metadata-uo9lo7si/python_lorem-1.3.0.post1.dist-info'
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 368, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 313, in run_setup
exec(code, locals())
File "", line 200, in
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 184, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
dist.run_commands()
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 976, in run_command
super().run_command(command)
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/command/dist_info.py", line 97, in run
bdist_wheel = self.get_finalized_command('bdist_wheel')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 302, in get_finalized_command
cmd_obj = self.distribution.get_command_obj(command, create)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 862, in get_command_obj
cmd_obj = self.command_obj[command] = klass(self)
^^^^^^^^^^^
TypeError: 'NoneType' object is not callable
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
[2024-06-21T14:34:41.442Z] error: metadata-generation-failed
× Encountered error while generating package metadata.
╰─> See above for output.
note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
```
Closes #6
Build output for failed python-lorem installation
[72 lines of output] /tmp/pip-build-env-0z2zwjj3/overlay/lib/python3.12/site-packages/setuptools/dist.py:478: SetuptoolsDeprecationWarning: Invalid dash-separated options !! ******************************************************************************** Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead. By 2024-Sep-26, you need to update your project and remove deprecated calls or your builds will no longer be supported. See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details. ******************************************************************************** !! opt = self.warn_dash_deprecation(opt, section) running dist_info creating /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info writing /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/PKG-INFO writing dependency_links to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/dependency_links.txt writing requirements to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/requires.txt writing top-level names to /tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/top_level.txt writing manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt' reading manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no previously-included files matching '.DS_Store' found anywhere in distribution warning: no files found matching '*.rst' anywhere in distribution no previously-included directories found matching '.eggs' no previously-included directories found matching '.github' no previously-included directories found matching '.venv' no previously-included directories found matching 'docs' no previously-included directories found matching 'temp' adding license file 'LICENSE' writing manifest file '/tmp/pip-modern-metadata-uo9lo7si/python_lorem.egg-info/SOURCES.txt' creating '/tmp/pip-modern-metadata-uo9lo7si/python_lorem-1.3.0.post1.dist-info' Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in