mingwli commented 6 years ago

TODOs:

prerequisites:

[x] compare impact's docfx.yml with repo's docfx.json, .openpublishing.publish.config.json, .openpublishing.redirection.json => gather diffs
[x] classify configuration properties(based on property resource origin => e.g. from local file or a server-side resource? config property meaning(description, build behavior)
[x] figure out the source of the missing fields!!!!!

migration-tool steps:

[x] input: local repo
[ ] load default template, fill-in the template with existing configs that can be found from repo files
[ ] gathering missing fields
[ ] output: the fully filled docfx.yml
[ ] fix repo content, e.g. document with redirection_url header
[ ] creating pull-request

migration CI build

[ ] 1. Azure DevOps CI template
[ ] 2. Config a OPS CI service
[ ] 3. Usage: configure repo specific CI job from OPS portal, this job will use CI service configured in step 2 and clone a Azure DevOps CI job with addition paramter.

mingwli commented 6 years ago

Some related APIs:

GET DocsetConfig API: /v1/Configurations?git_repo_url=https://github.com/Microsoft/azure-docs-pr&branch=master(which returns exactly the .openpublishing.publish.config.json) sample
Sample Response

{
  "build_entry_point": "docs",
  "docsets_to_publish": [
    {
      "docset_name": "azure-documents",
      "build_source_folder": ".",
      "build_output_subfolder": "azure",
      "locale": "en-us",
      "monikers": [],
      "moniker_ranges": [],
      "open_to_public_contributors": true,
      "type_mapping": {
        "Conceptual": "Content",
        "ManagedReference": "Content",
        "LandingData": "Content",
        "RestApi": "Content",
        "ContextObject": "Toc"
      },
      "build_entry_point": "docs",
      "template_folder": "_themes",
      "version": 0
    }
  ],
  "notification_subscribers": [],
  "branches_to_filter": [],
  "git_repository_url_open_to_public_contributors": "https://github.com/Microsoft/azure-docs",
  "git_repository_branch_open_to_public_contributors": "master",
  "skip_source_output_uploading": false,
  "need_preview_pull_request": true,
  "contribution_branch_mappings": {},
  "dependent_repositories": [
    {
      "path_to_root": "_themes",
      "url": "https://github.com/Microsoft/templates.docs.msft",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "_themes.pdf",
      "url": "https://github.com/Microsoft/templates.docs.msft.pdf",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "cli_scripts",
      "url": "https://github.com/Azure/azure-docs-cli-python-samples",
      "branch": "master",
      "branch_mapping": {
        "release-build-mysql": "release-build",
        "release-build-postgresql": "release-build",
        "release-build-stellar": "release-build"
      }
    },
    {
      "path_to_root": "powershell_scripts",
      "url": "https://github.com/Azure/azure-docs-powershell-samples",
      "branch": "master",
      "branch_mapping": {
        "release-build-mysql": "release-build",
        "release-build-postgresql": "release-build",
        "release-build-stellar": "release-build"
      }
    },
    {
      "path_to_root": "policy-templates",
      "url": "https://github.com/Azure/azure-policy",
      "branch": "master",
      "branch_mapping": {
        "release-build-mysql": "release-build",
        "release-build-postgresql": "release-build",
        "release-build-stellar": "release-build"
      }
    },
    {
      "path_to_root": "azure-docs-json-samples",
      "url": "https://github.com/Azure/azure-docs-json-samples",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-integration",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-functions-integration",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-encryptiondrm",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-dynamic-encryption-with-drm",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-encryptionfairplay",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-dynamic-encryption-with-fairplay",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-encryptionaes",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-dynamic-encryption-with-aes",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-copyblob",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-copy-blob-into-asset",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-deliverplayready",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-deliver-playready-widevine-licenses",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-livestream",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-encode-live-stream-with-ams-clear",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-mediaservices-encoderstandard",
      "url": "https://github.com/Azure-Samples/media-services-dotnet-on-demand-encoding-with-media-encoder-standard",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "azure-app-service-multi-container",
      "url": "https://github.com/Azure-Samples/multicontainerwordpress",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-durable-functions",
      "url": "https://github.com/Azure/azure-functions-durable-extension",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-luis",
      "url": "https://github.com/Microsoft/Luis-Samples",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "api-management-policy-samples",
      "url": "https://github.com/Azure/api-management-policy-snippets",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "WebApp-OpenIdConnect-DotNet",
      "url": "https://github.com/AzureADQuickStarts/WebApp-OpenIdConnect-DotNet",
      "branch": "GuidedSetup",
      "branch_mapping": {}
    },
    {
      "path_to_root": "iot-samples-node",
      "url": "https://github.com/Azure-Samples/azure-iot-samples-node",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-cognitive-services-speech-sdk",
      "url": "https://github.com/Azure-Samples/cognitive-services-speech-sdk",
      "branch": "docs-201807",
      "branch_mapping": {}
    },
    {
      "path_to_root": "media-services-v3-dotnet-quickstarts",
      "url": "https://github.com/Azure-Samples/media-services-v3-dotnet-quickstarts",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "media-services-v3-dotnet-tutorials",
      "url": "https://github.com/Azure-Samples/media-services-v3-dotnet-tutorials",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "samples-javascript",
      "url": "https://github.com/Microsoft/tsiclient",
      "branch": "tutorial",
      "branch_mapping": {}
    },
    {
      "path_to_root": "media-services-v3-dotnet-core-tutorials",
      "url": "https://github.com/Azure-Samples/media-services-v3-dotnet-core-tutorials",
      "branch": "master",
      "branch_mapping": {}
    },
    {
      "path_to_root": "media-services-v3-rest-postman",
      "url": "https://github.com/Azure-Samples/media-services-v3-rest-postman",
      "branch": "master",
      "branch_mapping": {}
    }
  ],
  "branch_target_mapping": {
    "live": [
      "Publish",
      "PDF"
    ],
    "master": [
      "Publish",
      "PDF"
    ],
    "release-event-grid": [
      "Publish",
      "PDF"
    ],
    "hd-insight-pdf": [
      "Publish",
      "PDF"
    ]
  },
  "need_generate_pdf_url_template": true,
  "Targets": {
    "Pdf": {
      "template_folder": "_themes.pdf"
    }
  }
}

GET DocfxConfig API: /v1/Configurations/docfx?git_repo_url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fazure-docs-pr&branch=master

which seems that .openpublishing.publish.config.json is a combination of the following two APIs.

GET DocsetInfo Detail API: /v2/Repositories/{repository_id}/Docsets/{docset_id} sample
Sample Response

{
    "build_source_folder": ".",
    "build_output_subfolder": "azure",
    "monikers": [],
    "moniker_ranges": [],
    "build_entry_point": "docs",
    "open_to_public_contributors": true,
    "type_mapping": {
        "Conceptual": "Content",
        "ManagedReference": "Content",
        "LandingData": "Content",
        "RestApi": "Content",
        "ContextObject": "Toc"
    },
    "template_folder": "_themes",
    "id": "e740e30b-9737-e0e5-192d-ae18c5d5258f",
    "name": "azure-documents",
    "product_name": "Azure",
    "is_active": true,
    "api_scan": false,
    "site_name": "Docs",
    "area": "",
    "theme": "Docs.Theme",
    "base_path": "/azure",
    "locale": "en-us",
    "status": "Created",
    "tenant": "c+e",
    "priority": 0,
    "repository_id": "8257e7f2-de62-9622-af05-1281f54738ea",
    "is_dynamic_rendering": true,
    "use_template": false
}

GET Docsets Info from repo: /v2/Queries/Docsets/git_repo_url={url}&docset_query_status={Created} sample

Sample Response

```yml [ { "id": "0f609cb0-c5dd-f173-1744-9b3723f23d64", "name": "adm", "product_name": "MSDN", "is_active": true, "api_scan": true, "site_name": "Docs", "area": "", "theme": "Docs.Theme", "base_path": "/dotnet", "locale": "en-us", "status": "Created", "tenant": "c+e", "priority": 70, "repository_id": "264ae87f-b6a7-7aad-b38a-c09700898eed", "is_dynamic_rendering": true, "use_template": false }, { "id": "7e8a0870-3494-4288-281d-cc8428805815", "name": "ADALRef_DocsV2", "product_name": "Azure", "is_active": true, "api_scan": false, "site_name": "Docs", "area": "", "theme": "Docs.Theme", "base_path": "/active-directory/adal/v2", "locale": "en-us", "status": "Created", "tenant": "c+e", "priority": 0, "repository_id": "264ae87f-b6a7-7aad-b38a-c09700898eed", "is_dynamic_rendering": true, "use_template": false }, { "id": "82f84d4d-1d70-babb-dfaf-8a097e9f0b58", "name": "ADALRef_Docs", "product_name": "Azure", "is_active": true, "api_scan": false, "site_name": "Docs", "area": "", "theme": "Docs.Theme", "base_path": "/active-directory/adal", "locale": "en-us", "status": "Created", "tenant": "c+e", "priority": 0, "repository_id": "264ae87f-b6a7-7aad-b38a-c09700898eed", "is_dynamic_rendering": true, "use_template": false }, { "id": "f8089889-d857-78eb-1776-1ecfbd0f6083", "name": "ADALRef_DocsV3", "product_name": "Azure", "is_active": true, "api_scan": false, "site_name": "Docs", "area": "", "theme": "Docs.Theme", "base_path": "/active-directory/adal/v3", "locale": "en-us", "status": "Created", "tenant": "c+e", "priority": 0, "repository_id": "264ae87f-b6a7-7aad-b38a-c09700898eed", "is_dynamic_rendering": true, "use_template": false } ] ```

Fields to treat

Field To Treat	Issues with Field	Actions
`name`	missing	in `.openpublishing.publish.config.json: docset_name` or `DocsetInfo Detail API: name`
`product`	missing	in `DocsetInfo Detail API: product_name`
`content`	can not use directly
`siteBasePath`	missing	`DocsetInfo Detail API: base_path` normalized
`followRedirect`	new in v3	???
`baseUrl`	missing	???
`needGeneratePdfUrlTemplate`	naming convention changed	`docfx.json: need_generate_pdf_url_template`
`documentId`	naming convention changed	`docfx.json: _op_documentIdPathDepotMapping`
`globalMetadata`	schema structure changed	`docfx.json: globalMetadata`
`routes`	new in v3
`rules`	new in v3	need default settings
`contribution`		partially in `docfx.json globalMetadata`
`dependencies`		in `.openpublishing.publish.config.json: dependent_repositories`
`fileMetadata`	schema structure changed	`docfx.json: fileMetadata`
`redirections`		`.openpublishing.redirection.json: redirections`

OsmondJiang commented 6 years ago

@mingwli we don't need migrate the API contract, actually there are many API contracts, we just need migrate from docfx.json + ops.config.json => docfx.yml

mingwli commented 6 years ago

@OsmondJiang Yes, thanks for reminding. I'm just trying to see if there is any useful information in these APIs. e.g. product field doesn't exist in neither .openpublishing.publish.config.json' nordocfx.json`.

mingwli commented 6 years ago

Field list:

[x] name: different per docset
[x] product: different per docset
[x] content
[x] siteBasePath: different per docset(docfx.json => dest)
[x] followRedirect
[x] baseUrl
[x] needGeneratePdfUrlTemplate
[x] documentId: need product_name from OPS server
[x] globalMetadata: different per docset, can use fileMetadata(resonable)
[x] routes
[x] rules
[x] contribution
[x] dependencies
[x] fileMetadata: different per docset
[x] redirections

OsmondJiang commented 6 years ago

3448 and #3449

OsmondJiang commented 5 years ago

I am closing this issue now, will create another issue to tracking current migration tool issues.

dotnet / docfx

migration tool to enable v3 for v2 repo #2999

TODOs:

prerequisites:

migration-tool steps:

migration CI build

Some related APIs:

Fields to treat

3448 and #3449