Remove executor id/secret caching

[nomagick]

As we are deprecating the secret and introducing namespacing

The id and secret caching mechanism are causing more trouble than it's trying to solve:

• After removing of a particular executor, the id is always cached, and users cannot re-push the executor from the same directory.
• After copy-pasting the executor directory on the harddrive, jina hub push defaults to executor creation, which triggers NamespaceOccupiedError.

Proposals:

For SDK:

• [x] Remove local executor id/secret caching feature completely
• [x] Use name from config.yml/manifest.yml as id.
• [ ] config.yml and name in it are required for executors just like package.json in npm
• [ ] Throws an error if the executor doesn't have a name or config.yml
• [ ] Provide jina hub init to initialize just the config.yml.

For Hubble backend:

• [ ] Automatically fall back to executor create when calling the update API. https://github.com/jina-ai/hubble/pull/766
• [ ] Query executor async tasks. https://github.com/jina-ai/hubble/pull/779

User Journey: (Without all-in-one push API)

1. The user creates a directory which contains executor files. Which may or may not contain config.yml.
2. The user runs jina hub push . in the executor directory
3. The SDK was invoked to process the request
4. The SDK parses name from config.yml/manifest.yml in the directory and throws an error if config.yml/manifest.yml is not found or name cannot be fetched.
5. In the error message, it suggests the user run jina hub init . to create a config.yml file automatically.
6. User runs jina hub init .
7. The SDK trys to load config.yml. If it cannot find a name in config.yml, use the directory name as name. Then writes config.yml to disk.
8. User runs jina hub push . again
9. The SDK calls Hubble API executor.listAsyncTasks with parameter id=[name] to check for executor status.
10. If the API returns with HTTP Code 404, It means the executor doesn't exist. SDK should continue with the previous jina hub push .
11. If the API returns with HTTP Code 200, and with a truthy body meta blockOn. It means the executor is being created. The SDK should first wait for the previous job to finish. At this time, SDK should continue with previous jina hub status --id [blockOn id] || jina hub push .,
12. If the API returns with HTTP Code 200, and without a truthy body meta blockOn. It means the executor already exists and the SDK should continue with previous jina hub push . --force [name]
13. User runs jina hub status .
14. The SDK parses the executor name as described above
15. The SDK calls Hubble API executor.listAsyncTasks with parameter id=[name], sort=-createdAt to check for executor status.
16. The SDK Continue with previous jina hub status --id [n] where n is the _id of the first returned record. Throws error otherwise for no status to query.

[mapleeit]

I agree with all the items except:

Provide jina hub init to initialize just the config.yml.

We already have jina hub new which serves for this purpose. So maybe we can remove this item?

[nomagick]

jina hub new creates a new directory and multiple files, while jina hub init is meant to only create config.yml in existing directory. I think it might be useful for people doing the migration after they receive "config.yml required" error.

[github-actions[bot]]

:tada: This issue has been resolved in version 0.26.12 :tada:

The release is available on:

• v0.26.12
• GitHub release

Your semantic-release bot :package::rocket: