Refactor DA outputs - Githubissues

rajatagarwal-ibm commented 6 months ago

Rename:

project_id -> watsonx_project_id project_url -> watsonx_project_url watsonx_assistant_url -> watsonx_assistant_api_url watsonx_discovery_url -> watsonx_discovery_api_url

Create new outputs:

watsonx_assistant_dashboard_url
watsonx_discovery_dashboard_url

rajatagarwal-ibm commented 6 months ago

TLDR - We don't have CRN for assistant and discovery projects which is required for the dashboard URL. One option is to use RestAPI like configure_project did in the SaaS DA. The RestAPI object returns a lot of values (see below), including CRN.

Assistant Dashboard URL:

Sample assistant dashboard URL - https://us-south.assistant.watson.cloud.ibm.com/crn%3Av1%3Abluemix%3Apublic%3Aconversation%3Aus-south%3Aa%2Fabac0df06b644a9cabc6e44f55b3880e%3A9bdd9a67-995c-4a82-80ea-edc005a34d4e%3A%3A/assistants/1ed310a1-ace5-4e1a-8a7a-48ce7cb3eaba/home

URL components are:

Hostname - It's available. Can tweak it based on the region supplied (currently everything is in us-south anyway).
CRN - NOT AVAILABLE. As per the Assistant API docs, there is no provision to return the CRN. See https://cloud.ibm.com/apidocs/assistant-v2#listassistants
Assistant Project ID: It's available.

Discovery Dashboard URL:

Sample discovery dashboard URL - https://us-south.discovery.watson.cloud.ibm.com/v2/instances/crn%3Av1%3Abluemix%3Apublic%3Adiscovery%3Aus-south%3Aa%2Fabac0df06b644a9cabc6e44f55b3880e%3Ab96e09cf-3d44-4a02-80e9-1762c613f1ca%3A%3A/projects

URL components are:

Hostname - It's available. Can tweak it based on the region supplied (currently everything is in us-south anyway).
CRN - NOT AVAILABLE. As per the Discovery API docs, there is no provision to return the CRN. See https://cloud.ibm.com/apidocs/discovery-data#listprojects

Note: Configure project uses the "restapi" provider for set up, but we use scripts. This is the object returned by the Rest API provider

crn = {
  "api_data" = tomap({
    "entity" = <REDACTING AS CONTAINS API KEY>
  })
  "api_response" = <REDACTING AS CONTAINS SECRET KEYS>
  "create_method" = "POST"
  "create_path" = "//api.dataplatform.cloud.ibm.com/transactional/v2/projects?verify_unique_name=true"
  "create_response" = "{\"location\":\"/v2/projects/c37a1b40-fb7f-406a-97b7-e27b45dba722\"}"
  "data" = <<-EOT
  {
    "name": "rag-del-test-RAG-sample-project",
    "generator": "watsonx-saas-da",
    "type": "wx",
    "storage": {
      "type": "bmcos_object_storage",
      "guid": "b1a4e4a0-e23e-413d-bc98-b4e894f85827",
      "resource_crn": "crn:v1:bluemix:public:cloud-object-storage:global:a/abac0df06b644a9cabc6e44f55b3880e:b1a4e4a0-e23e-413d-bc98-b4e894f85827::"
    },
    "description": "WatsonX AI project for RAG pattern sample app",
    "public": true,
    "tags": ["watsonx-ai-SaaS","RAG-sample-project"],
    "compute": [
      {
        "name": "rag-del-watson-machine-learning-instance",
        "guid": "85dd3375-b94a-4d02-abeb-867d1ca5b376",
        "type": "machine_learning",
        "crn": "crn:v1:bluemix:public:pm-20:us-south:a/abac0df06b644a9cabc6e44f55b3880e:85dd3375-b94a-4d02-abeb-867d1ca5b376::"
      }
    ]
  }

  EOT
  "debug" = tobool(null)
  "destroy_data" = tostring(null)
  "destroy_method" = "DELETE"
  "destroy_path" = "//api.dataplatform.cloud.ibm.com/transactional{id}"
  "force_new" = tolist(null) /* of string */
  "id" = "/v2/projects/c37a1b40-fb7f-406a-97b7-e27b45dba722"
  "id_attribute" = "location"
  "object_id" = tostring(null)
  "path" = "//api.dataplatform.cloud.ibm.com"
  "query_string" = tostring(null)
  "read_data" = tostring(null)
  "read_method" = "GET"
  "read_path" = "//api.dataplatform.cloud.ibm.com{id}"
  "read_search" = tomap(null) /* of string */
  "update_data" = <<-EOT
  {
    "name": "rag-del-test-RAG-sample-project",
    "type": "wx",
    "description": "WatsonX AI project for RAG pattern sample app",
    "public": true,
    "compute": [
      {
        "name": "rag-del-watson-machine-learning-instance",
        "guid": "85dd3375-b94a-4d02-abeb-867d1ca5b376",
        "type": "machine_learning",
        "crn": "crn:v1:bluemix:public:pm-20:us-south:a/abac0df06b644a9cabc6e44f55b3880e:85dd3375-b94a-4d02-abeb-867d1ca5b376::",
        "credentials": {}
      }
    ]
  }

  EOT
  "update_method" = "PATCH"
  "update_path" = "//api.dataplatform.cloud.ibm.com{id}"
}

rajatagarwal-ibm commented 6 months ago

Discovery Rest API testing:

Tested with the following piece of code to create a Discovery project using the RestAPI (just like SaaS DA)

resource "restapi_object" "discovery_project_creation_api" {
  depends_on       = [data.ibm_iam_auth_token.tokendata]
  path                    = local.watsonx_discovery_url
  create_path       = "${local.watsonx_discovery_url}/v2/projects?version=2023-03-31"
  create_method  = "POST"
  id_attribute        = "project_id"
  data                    = <<-EOT
      { "name": "gen-ai-rag-sample-app-project-12",   "type": "document_retrieval" }
  EOT
}

This is the output of the restapi object:

discovery_output = {
  "api_data" = tomap({
    "collection_count" = "0"
    "default_query_parameters" = "map[aggregation:[term(enriched_text.entities.text,name:entities)] count:10 highlight:false passages:map[characters:200 count:10 enabled:true fields:[text title] find_answers:false max_answers_per_passage:1 max_per_document:1 per_document:true] return:[] sort: spelling_suggestions:false table_results:map[count:10 enabled:false per_document:0]]"
    "name" = "gen-ai-rag-sample-app-project-12"
    "project_id" = "1753643f-1cb3-4845-ad7c-e17a6641887a"
    "type" = "document_retrieval"
  })
  "api_response" = <<-EOT
  {
    "project_id" : "1753643f-1cb3-4845-ad7c-e17a6641887a",
    "type" : "document_retrieval",
    "name" : "gen-ai-rag-sample-app-project-12",
    "collection_count" : 0,
    "default_query_parameters" : {
      "aggregation" : "[term(enriched_text.entities.text,name:entities)]",
      "count" : 10,
      "sort" : "",
      "return" : [ ],
      "passages" : {
        "enabled" : true,
        "count" : 10,
        "fields" : [ "text", "title" ],
        "characters" : 200,
        "per_document" : true,
        "max_per_document" : 1,
        "find_answers" : false,
        "max_answers_per_passage" : 1
      },
      "highlight" : false,
      "spelling_suggestions" : false,
      "table_results" : {
        "enabled" : false,
        "count" : 10,
        "per_document" : 0
      }
    }
  }
  EOT
  "create_method" = "POST"
  "create_path" = "//api.us-south.discovery.watson.cloud.ibm.com/instances/b96e09cf-3d44-4a02-80e9-1762c613f1ca/v2/projects?version=2023-03-31"
  "create_response" = <<-EOT
  {
    "project_id" : "1753643f-1cb3-4845-ad7c-e17a6641887a",
    "type" : "document_retrieval",
    "name" : "gen-ai-rag-sample-app-project-12",
    "collection_count" : 0,
    "default_query_parameters" : {
      "aggregation" : "[term(enriched_text.entities.text,name:entities)]",
      "count" : 10,
      "sort" : "",
      "return" : [ ],
      "passages" : {
        "enabled" : true,
        "count" : 10,
        "fields" : [ "text", "title" ],
        "characters" : 200,
        "per_document" : true,
        "max_per_document" : 1,
        "find_answers" : false,
        "max_answers_per_passage" : 1
      },
      "highlight" : false,
      "spelling_suggestions" : false,
      "table_results" : {
        "enabled" : false,
        "count" : 10,
        "per_document" : 0
      }
    }
  }
  EOT
  "data" = <<-EOT
  { "name": "gen-ai-rag-sample-app-project-12",   "type": "document_retrieval" }

  EOT
  "debug" = tobool(null)
  "destroy_data" = tostring(null)
  "destroy_method" = tostring(null)
  "destroy_path" = tostring(null)
  "force_new" = tolist(null) /* of string */
  "id" = "1753643f-1cb3-4845-ad7c-e17a6641887a"
  "id_attribute" = "project_id"
  "object_id" = tostring(null)
  "path" = "//api.us-south.discovery.watson.cloud.ibm.com/instances/b96e09cf-3d44-4a02-80e9-1762c613f1ca"
  "query_string" = tostring(null)
  "read_data" = tostring(null)
  "read_method" = tostring(null)
  "read_path" = tostring(null)
  "read_search" = tomap(null) /* of string */
  "update_data" = tostring(null)
  "update_method" = tostring(null)
  "update_path" = tostring(null)
}

rajatagarwal-ibm commented 6 months ago

Issues opened within Assistant and Discovery repos

rajatagarwal-ibm commented 6 months ago

Closing as its being already tracked here https://github.com/terraform-ibm-modules/terraform-ibm-rag-sample-da/issues/21

terraform-ibm-modules / terraform-ibm-rag-sample-da

Refactor DA outputs #45