Publish service that imports common proto definition

ferrouswheel commented 5 years ago

I'm trying to publish one of the face-services.

All the face-services rely on a shared protobuf definition file face_common.proto.

When I try to publish the face detection service, I get this error:

$ snet service publish
Error: Import path must be a valid file: /home/joel/work/services/face-services/services/grpc/services/grpc/face_common.proto

This error is correct, there is no file at that location. It is duplicating the services/grpc part of the path.

However, whatever I try, I can't get it to use the correct path for the import.

I have tried:

changing the import statement in face_detect.proto to use a relative path to the file being processed
a relative path to the current working directory
ensuring face_common.proto in the same directory as face_detect.proto and using no path prefix: import "face_common.proto";

What ever I try, it always adds the prefix services/grpc twice instead of once.

The reason sharing a common definition is important is that then APIs will rely on consistent primitive message types. Without this, every API is essentially independent and application developers will have to spend all their time converting between types when relying on multiple SNet services.

ferrouswheel commented 5 years ago

Since this is, for now, only specific to my particular use case, I'm happy to try to figure this out. Before I do so, it'd be great to have some insight from others who have worked with the cli recently. cc @vforvalerio87 @arturgontijo

arturgontijo commented 5 years ago

Here I got it published using import "face_common.proto"; But I didn't test calling it...

(snet_016) artur@singNET:~/face-services/services$ 
cat grpc/face_detect.proto 
syntax = 'proto3';

import "face_common.proto";

service FaceDetect {
  rpc FindFace (stream ImageRGB) returns (FaceDetections) {}
(snet_016) artur@singNET:~/face-services/services$ 
snet service init
Please provide values to populate your service.json file

Choose a name for your service: (default: "services")
TestImportProto               
Choose the path to your service's spec directory: (default: "service_spec/")
grpc
Choose an organization to register your service under: (required)
SNET
Choose the path under which your Service registration will be created: (default: "")

Choose a price in AGI to call your service: (default: 0)
2
Endpoint to call the API for your service, should start with http(s):// : (default: "")
http://54.203.198.53:7003        
Input a list of tags for your service, space separated: (default: [])
1 2 3 4
Input a description for your service: (default: "")
1234
{
    "name": "TestImportProto",
    "service_spec": "grpc",
    "organization": "SNET",
    "path": "",
    "price": 2,
    "endpoint": "http://54.203.198.53:7003",
    "tags": [
        "1",
        "2",
        "3",
        "4"
    ],
    "metadata": {
        "description": "1234"
    }
}

service.json file has been created!
(snet_016) artur@singNET:~/face-services/services$ 
snet service publish

5 valid protobuf file(s) found.

Unreachable endpoint (should start with http(s)://): http://54.203.198.53:7003
Proceed? (y/n): y
Creating transaction to create agent contract...

    transaction:
        chainId: 42
        data: '...'
        from: '0xFF2a327ed1Ca40CE93F116C5d6646b56991c0ddE'
        gas: 1788459
        gasPrice: 1000000000
        nonce: 4139
        to: '0x1fAa8ec70aFe4f5ce904dA935A6ddF5f3482eEDb'
        value: 0

Proceed? (y/n): y
Submitting transaction...

    event_summaries:
    -   args:
            agent: '0xDdC6675bbcfBE7A2a82aD1DDE1aF94BF48C99935'
        event: AgentCreated
    receipt_summary:
        blockHash: '0x26b6d734d402af5dd3ed9eb105bcc8f9fc3710d8daf2455a7f15a240066ba0c5'
        blockNumber: 9171096
        cumulativeGasUsed: 1874569
        gasUsed: 1788285
        transactionHash: '0x64859fcb8321ff46afb514cf003c308a67158cbd132d7ec92e33915ef11596aa'

Adding contract address to service.json file...

Creating transaction to create service registration...

    transaction:
        chainId: 42
        data: '...'
        from: '0xFF2a327ed1Ca40CE93F116C5d6646b56991c0ddE'
        gas: 797511
        gasPrice: 1000000000
        nonce: 4140
        to: '0x440cF8424fcD7Fc2D2fF3a5668c919E93A3d2aAb'
        value: 0

Proceed? (y/n): y
Submitting transaction...

    event_summaries:
    -   args:
            orgName: 534e455400000000000000000000000000000000000000000000000000000000
            orgNameIndexed: 534e455400000000000000000000000000000000000000000000000000000000
            serviceName: 54657374496d706f727450726f746f0000000000000000000000000000000000
            serviceNameIndexed: 54657374496d706f727450726f746f0000000000000000000000000000000000
        event: ServiceCreated
    receipt_summary:
        blockHash: '0x07affe00a56138bbfae76ecea926738c40722539a044a571543efaa41a95941a'
        blockNumber: 9171097
        cumulativeGasUsed: 2280027
        gasUsed: 797511
        transactionHash: '0xff91e6a4045fff95383d3a998f4f28a10ddf4d81d25c2df007e4c2ff8702aa5c'

Adding contract address to session...

set current_agent_at 0xDdC6675bbcfBE7A2a82aD1DDE1aF94BF48C99935

Service published!
(snet_016) artur@singNET:~/face-services/services$ 
snet client get-spec GET_SPEC/
Retrieving service spec of 0xDdC6675bbcfBE7A2a82aD1DDE1aF94BF48C99935
    destination: GET_SPEC/

(snet_016) artur@singNET:~/face-services/services$  
ls -la GET_SPEC/
total 28
drwxrwxr-x 2 artur artur 4096 Out 24 08:11 .
drwxrwxr-x 4 artur artur 4096 Out 24 08:11 ..
-rw-rw-r-- 1 artur artur  707 Out 24 08:07 face_alignment.proto
-rw-rw-r-- 1 artur artur  600 Out 24 08:05 face_common.proto
-rw-rw-r-- 1 artur artur  132 Out 24 08:07 face_detect.proto
-rw-rw-r-- 1 artur artur  559 Out 24 08:07 face_landmarks.proto
-rw-rw-r-- 1 artur artur  625 Out 24 08:07 face_recognition.proto
(snet_016) artur@singNET:~/face-services/services$ 
cat GET_SPEC/face_detect.proto 
syntax = 'proto3';

import "face_common.proto";

service FaceDetect {
  rpc FindFace (stream ImageRGB) returns (FaceDetections) {}

ferrouswheel commented 5 years ago

@arturgontijo Ok, so maybe the issue is related to not having a nested path structure in service_spec?

You use grpc and run from the services directory. I was running from the root of the project and used services\grpc as the service_spec

ferrouswheel commented 5 years ago

I've managed to publish my face-services using the original config, so whatever problem I was having is no longer an issue! :sweat_smile:

ferrouswheel commented 5 years ago

I might have spoken too soon, see issue referencing this one in service-spec-provider. Looks like the error was just kicked down the pipeline :-(

ferrouswheel commented 5 years ago

Looks like we attempt to compile on snet channel open-init now, and the issue with paths returns:

$ snet channel open-init snet face-detect 0.000001 10279000
services/grpc/face_common.proto: File not found.
face_landmarks.proto: Import "services/grpc/face_common.proto" was not found or had errors.
face_landmarks.proto:7:5: "FaceDetections" is not defined.
face_landmarks.proto:13:9: "ImageRGB" is not defined.
face_landmarks.proto:18:14: "FaceLandmarks" is not defined.
face_landmarks.proto:25:44: "FaceLandmarkModels" is not defined.
Error: Fail to compile /tmp/tmph5y46hwq/service_spec/*.proto
If you want to see full Traceback then run:
snet --print-traceback [parameters]

Reopening until I manage to find something that works.

ferrouswheel commented 5 years ago

Fixed by https://github.com/singnet/face-services/commit/1b37ec859a3cb29a73944877c69b3ca10dd368a3

Due to various decisions, we need to assume a completely flat namespace for our proto includes.

singnet / snet-cli

Publish service that imports common proto definition #84