delete_gateway command of syndicate tool does not behave properly

[iychoi]

iychoi@demo1:~/appspot_ms$ syndicate delete_gateway ag_imicrobe
Help for 'delete_gateway':

   Delete a gateway.

   Positional arguments:
      name (str):
         The name of the gateway to delete.

   Returns:
      True on success.
      Raises an exception on error.

   Authorization:
      An administrator can delete any gateway.
      A volume oner can delete any gateway in the volumes (s)he owns.
      A user cannot delete the gateway.

   Remarks:
      Syndicate should generate 'cert_bundle_b64'
      as a keyword argument, which contains the base64-encoded serialized 
      cert bundle version vector for this volume.

iychoi@demo1:~/appspot_ms$ syndicate delete_gateway ag_imicrobe
true
iychoi@demo1:~/appspot_ms$ 

When the command issued first time, it returns usage (which generally means error) then it returns true for repeating commands.

[iychoi]

I observe this issue with other syndicate commands as well. It seems this happens when MS is not responding on time.

[iychoi]

The same issue is observed at appspot production MS. Not sure why this happens.

[iychoi]

[2017-06-06 00:07:06,598] [DEBUG] [client:270] (14231) As jcnelson@cs.princeton.
edu, call read_gateway(imicrobe_anonymous )
[2017-06-06 00:07:07,894] [DEBUG] [client:270] (14231) As jcnelson@cs.princeton$
edu, call delete_gateway(imicrobe_anonymous )
[2017-06-06 00:07:09,264] [ERROR] [storage:175] (14231) Failed to stat /home/iy$
hoi/.syndicate/gateways/imicrobe_anonymous.pkey
[2017-06-06 00:07:09,270] [ERROR] [syndicate:1138] (14231) No volume cert for '$
053698068735486726'
Traceback (most recent call last):
  File "/usr/bin/syndicate", line 1135, in main
    result = client.ms_rpc( rpc_client, method_name, *args, **kw )
  File "/usr/lib/python2.7/dist-packages/syndicate/util/client.py", line 276, i$
 ms_rpc
    object_cls.PostProcessResult( extras, config, method_name, args, kw, ret )
  File "/usr/lib/python2.7/dist-packages/syndicate/util/objects.py", line 2726, 
in PostProcessResult
    failed = do_volume_reload( config, reload_owner_id, volume_id )
  File "/usr/lib/python2.7/dist-packages/syndicate/util/objects.py", line 975, $
n do_volume_reload
    statuses = reloader.broadcast_reload( config, user_id, volume_id )
  File "/usr/lib/python2.7/dist-packages/syndicate/util/reload.py", line 371, in
 broadcast_reload
    raise MissingCertException("No volume cert for '%s'" % str(volume_id))
MissingCertException: No volume cert for '7053698068735486726'
Help for 'delete_gateway':
...

Logs.

[iychoi]

When a gateway is deleted, volume-reload is called in the post-process. If any of volume cert or volume owner's cert is missing, the issue happens. It needs to reload certs recursively.