AMF3 connect failed and decode error

[lgjx123]

1. src/main/java/org/red5/client/net/rtmp/BaseRTMPClientHandler.java onCommand method

   log.trace("onCommand: {}, id: {}", command, command.getTransactionId());
   final IServiceCall call = command.getCall();
   final String methodName = call.getServiceMethodName();
   log.debug("Service name: {} args[0]: {}", methodName, (call.getArguments().length != 0 ? call.getArguments()[0] : ""));
   if ("_result".equals(methodName) || "_error".equals(methodName)) {
      final IPendingServiceCall pendingCall = conn.getPendingCall(command.getTransactionId());
      log.debug("Received result for pending call - {}", pendingCall);
      if (pendingCall != null) {
          if ("connect".equals(methodName)) {
              Integer encoding = (Integer) connectionParams.get("objectEncoding");
              if (encoding != null && encoding.intValue() == 3) {
                  log.debug("Setting encoding to AMF3");
                  conn.getState().setEncoding(IConnection.Encoding.AMF3);
              }
          }
      }
      handlePendingCallResult(conn, (Invoke) command);
      return;
   }

   if ("connect".equals(methodName)) is unreach code, it should be if ("connect".equals(pendingCall.getServiceMethodName()))

2. server use these code to send data to client （org.red5.server.net.rtmp.codec.RTMPProtocolEncoder encodeCommand(IoBuffer out, ICommand command)）

   Output output = new org.red5.io.amf.Output(out);
   final IServiceCall call = command.getCall();
   final boolean isPending = (call.getStatus() == Call.STATUS_PENDING);
   log.debug("Call: {} pending: {}", call, isPending);
   if (!isPending) {
      log.debug("Call has been executed, send result");
      Serializer.serialize(output, call.isSuccess() ? "_result" : "_error");
   } else {
      log.debug("This is a pending call, send request");
      final String action = (call.getServiceName() == null) ? call.getServiceMethodName() : call.getServiceName() + '.' + call.getServiceMethodName();
      Serializer.serialize(output, action); // seems right
   }
   if (command instanceof Invoke) {
      Serializer.serialize(output, Integer.valueOf(command.getTransactionId()));
      Serializer.serialize(output, command.getConnectionParams());
   }

   and decode data like this( org.red5.server.net.rtmp.codec.RTMPProtocolDecoder decodeInvoke(Encoding encoding, IoBuffer in) )

   if (action != null) {
      Invoke invoke = new Invoke();
      invoke.setTransactionId(Deserializer.<Number> deserialize(input, Number.class).intValue());
      // now go back to the actual encoding to decode parameters
      if (encoding == Encoding.AMF3) {
          input = new org.red5.io.amf3.Input(in);
          ((org.red5.io.amf3.Input) input).enforceAMF3();
      } else {
          input = new org.red5.io.amf.Input(in);
      }
      // get / set the parameters if there any
      Object[] params = in.hasRemaining() ? handleParameters(in, invoke, input) : new Object[0];

   red5-client will thrown decode error cause Serializer.serialize(output, command.getConnectionParams());(still AMF0 but skip) when use AMF3.

   Now I try to override decodeInvoke method in RTMPClientProtocolDecoder ` @Override public Invoke decodeInvoke(Encoding encoding, IoBuffer in) {

    // for response, the action string and invokeId is always encoded as AMF0 we use the first byte to decide which encoding to use
    in.mark();
    byte tmp = in.get();
    in.reset();
    Input input;
    if (encoding == Encoding.AMF3 && tmp == AMF.TYPE_AMF3_OBJECT) {
        input = new org.red5.io.amf3.Input(in);
        ((org.red5.io.amf3.Input) input).enforceAMF3();
    } else {
        input = new org.red5.io.amf.Input(in);
    }
    // get the action
    String action = Deserializer.deserialize(input, String.class);
    if (log.isTraceEnabled()) {
        log.trace("Action {}", action);
    }
    // throw a runtime exception if there is no action
    if (action != null) {
        Invoke invoke = new Invoke();
        invoke.setTransactionId(Deserializer.<Number> deserialize(input, Number.class).intValue());
        // get / set the parameters if there any
        Object[] params = in.hasRemaining() ? clientHandleParameters(encoding,in, invoke, input) : new Object[0];
        // determine service information
        final int dotIndex = action.lastIndexOf('.');
        String serviceName = (dotIndex == -1) ? null : action.substring(0, dotIndex);
        // pull off the prefixes since java doesnt allow this on a method name
        if (serviceName != null && (serviceName.startsWith("@") || serviceName.startsWith("|"))) {
            serviceName = serviceName.substring(1);
        }
        String serviceMethod = (dotIndex == -1) ? action : action.substring(dotIndex + 1, action.length());
        // pull off the prefixes since java doesn't allow this on a method name
        if (serviceMethod.startsWith("@") || serviceMethod.startsWith("|")) {
            serviceMethod = serviceMethod.substring(1);
        }
        PendingCall call = new PendingCall(serviceName, serviceMethod, params);
        invoke.setCall(call);
        return invoke;
    } else {
        // TODO replace this with something better as time permits
        throw new RuntimeException("Action was null");
    }

   } `

` /* * Sets incoming connection parameters and / or returns encoded parameters for use in a call. * * @param in * @param notify * @param input * @return parameters array / private Object[] clientHandleParameters(Encoding encoding,IoBuffer in, Notify notify, Input input) {

    Object[] params = new Object[] {};
    List<Object> paramList = new ArrayList<Object>();
    final Object obj = Deserializer.deserialize(input, Object.class);
    if (obj instanceof Map) {
        // Before the actual parameters we sometimes (connect) get a map of parameters, this is usually null, but if set should be
        // passed to the connection object.
        @SuppressWarnings("unchecked")
        final Map<String, Object> connParams = (Map<String, Object>) obj;
        notify.setConnectionParams(connParams);
    } else if (obj != null) {
        paramList.add(obj);
    }

    // now go back to the actual encoding to decode parameters
    if (encoding == Encoding.AMF3) {
        input = new org.red5.io.amf3.Input(in);
        ((org.red5.io.amf3.Input) input).enforceAMF3();
    } else {
        input = new org.red5.io.amf.Input(in);
    }

    while (in.hasRemaining()) {
        paramList.add(Deserializer.deserialize(input, Object.class));
    }
    params = paramList.toArray();
    if (log.isDebugEnabled()) {
        log.debug("Num params: {}", paramList.size());
        for (int i = 0; i < params.length; i++) {
            log.debug(" > {}: {}", i, params[i]);
        }
    }
    return params;
}

` and it fixed the decode error.

[mondain]

I think some of the formatting is messed up in your post; its a little hard to follow.

[solomax]

Maybe this might be filed as PR with Test and the fix?

[mondain]

Test your code against 1.0.8-M8, it has AMF decoding fixes

[solomax]

client still unable to connect to RTMPS in tunneled mode :( going to check native later

On Tue, Aug 16, 2016 at 1:31 AM, Paul Gregoire notifications@github.com wrote:

Test your code against 1.0.8-M8, it has AMF decoding fixes

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Red5/red5-client/issues/35#issuecomment-239886903, or mute the thread https://github.com/notifications/unsubscribe-auth/ADsPf3O7XB9Xb9sdp5DhGwKejByFcDdmks5qgLCfgaJpZM4JaNUV .

WBR Maxim aka solomax