RasaHQ / rasa-x-helm

Rasa Enterprise Helm chart for deploying on Kubernetes (K8s) and OpenShift.
Apache License 2.0
75 stars 104 forks source link

Cannot upload model to rasa x 1.1 #289

Open nyejon opened 2 years ago

nyejon commented 2 years ago
    return await await_and_return_response(args, kwargs, request)
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 176, in await_and_return_response
    response = await response
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/decorators.py", line 84, in decorated_function
    return await f(request, user=user, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/api/blueprints/models.py", line 69, in upload_model
    minimum_version = await model_service.minimum_compatible_version()
  File "/usr/local/lib/python3.8/dist-packages/rasax/community/services/model_service.py", line 208, in minimum_compatible_version
    info = await stack_service.version()
AttributeError: 'NoneType' object has no attribute 'version'
INFO:sanic.access:
willymwai commented 2 years ago

Was facing a similar problem, I solved it by enabling rasaProduction in the helm chart. It's false by default. When enabled, it adds a production url to environments.yml

sara-tagger commented 2 years ago

Thanks for raising this issue, @hsm207 will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗
nyejon commented 2 years ago

Why was this closed?

https://github.com/RasaHQ/rasa-x-helm/blob/68b36820918c26074fe662491150e821baf64d20/charts/rasa-x/values.yaml#L223-L235

States that we should be able to disable the rasa production deployment if we use an external server. I have populated the external server url, and set it to true.

I am trying to use the rasa-oss chart to deploy the models as suggested, but so much seems broken/inconsistent between them.

nyejon commented 2 years ago

From the helm chart:

# rasa: Settings common for all Rasa containers
# deprecated: the Rasa OSS deployment is deprecated and will be removed in the feature
# from this chart.
# It's recommended to use the rasa helm chart instead.
# see: https://github.com/RasaHQ/helm-charts/tree/main/charts/rasa#quick-start
hsm207 commented 2 years ago

I am trying to use the rasa-oss chart to deploy the models as suggested, but so much seems broken/inconsistent between them.

@virtualroot could you please look into this?

virtualroot commented 2 years ago

@nyejon Rasa Enterprise 1.1 is no longer compatible with Rasa Open Source 2. Is it possible the model was made with Rasa 2.x?

nyejon commented 2 years ago

No, I am trying to upload a freshly trained model with rasa 3.0.9.

nyejon commented 2 years ago

image

Nothing is populated in the environments as well.

nyejon commented 2 years ago

I would guess there is something wrong with this line: https://github.com/RasaHQ/rasa-x-helm/blob/68b36820918c26074fe662491150e821baf64d20/charts/rasa-x/templates/_rasa.tpl#L3

Also, with the logic from the charts, when using rasa-x as the config endpoint, should all the values for the rasa-oss deployment be populated in the rasa-x helm chart or the rasa-oss helm chart?

For example, the tracker-database name -> According to the rasa-X chart, the rasa config is deprecated, but according to the rasa-oss chart, the config is pulled from rasa-x

https://github.com/RasaHQ/helm-charts/blob/fb898a2613a4073b2a0233891df62e7db1e2da1a/charts/rasa/values.yaml#L59

This would apply to the rasa worker too. I don't see a way to specify from the Rasa OSS deployment, if it should pull production config from rasa x or the worker config.

nyejon commented 2 years ago

ok, I seem to have gotten something more working. It now populates the environments if I enable: rasaWorker: external: enabled: true

tczekajlo commented 2 years ago

I would guess there is something wrong with this line:

https://github.com/RasaHQ/rasa-x-helm/blob/68b36820918c26074fe662491150e821baf64d20/charts/rasa-x/templates/_rasa.tpl#L3

Also, with the logic from the charts, when using rasa-x as the config endpoint, should all the values for the rasa-oss deployment be populated in the rasa-x helm chart or the rasa-oss helm chart?

No, using Rasa X as a config endpoint for Rasa OSS makes that Rasa OSS knows about Tracker Store, Lock Store, and so on, but it doesn't mean that Rasa X knows about Rasa OSS.

External deployments for the production or worker environment have to be defined in the values for Rasa X.

Example:

rasa:
  token: rasaToken
  versions:
    rasaProduction:
      enabled: false
      external:
        enabled: true
        url: "http://rasa:5005" # address of an external Rasa OSS deployment
nyejon commented 2 years ago

External deployments for the production or worker environment have to be defined in the values for Rasa X.

Example:

I have all of that configured. How would I let a worker know which tracker database to use?

I can now get the OSS deployment to try and load a model I uploaded and was trained with 3.0.9. This results in the following error:

  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/graph.py", line 390, in _load_component
    self._component: GraphComponent = constructor(  # type: ignore[no-redef]
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1089, in load
    return cls._load(
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1115, in _load
    ) = cls._load_from_files(model_path)
  File "/opt/venv/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1152, in _load_from_files
    data_example = io_utils.pickle_load(
  File "/opt/venv/lib/python3.8/site-packages/rasa/utils/io.py", line 104, in pickle_load
    return pickle.load(f)
ModuleNotFoundError: No module named 'scipy.sparse._coo'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/agent.py", line 86, in _update_model_from_server
    _load_and_set_updated_model(agent, temporary_directory, new_fingerprint)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/agent.py", line 69, in _load_and_set_updated_model
    agent.load_model(model_directory, fingerprint)
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/agent.py", line 352, in load_model
    self.processor = MessageProcessor(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 96, in __init__
    self.model_filename, self.model_metadata, self.graph_runner = self._load_model(
  File "/opt/venv/lib/python3.8/site-packages/rasa/core/processor.py", line 121, in _load_model
    metadata, runner = loader.load_predict_graph_runner(
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/loader.py", line 29, in load_predict_graph_runner
    runner = graph_runner_class.create(
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 51, in create
    return cls(graph_schema, model_storage, execution_context, hooks)
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 37, in __init__
    self._instantiated_nodes: Dict[Text, GraphNode] = self._instantiate_nodes(
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 60, in _instantiate_nodes
    return {
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 61, in <dictcomp>
    node_name: GraphNode.from_schema_node(
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/graph.py", line 528, in from_schema_node
    return cls(
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/graph.py", line 379, in __init__
    self._load_component()
  File "/opt/venv/lib/python3.8/site-packages/rasa/engine/graph.py", line 402, in _load_component
    raise GraphComponentException(
rasa.engine.exceptions.GraphComponentException: Error initializing graph component for node 'run_DIETClassifier7'.

Edit, found this: https://github.com/RasaHQ/rasa/issues/10908