Blockwise with CoAP GET and receive file is too large

[iot-cloud]

Please providing basic issue information

Issues may be answered much faster, if you provide more details ahead!

• branch your faced the issue. Currently
  • 1.0 is the release,
  • 1.1 is the master,
  • and 2.0 is the development branch * If you use a SNAPSHOT, ensure your updated to the latest. Please provide the commit your using
• If your issue is related to your own snippet, please provide that
• If you have logs, please provide them
• If you have tcpdump captures (wireshark), please provide them.

I test out the firmware update approach using the simple file server at https://github.com/eclipse/californium/tree/2.0.x/demo-apps/cf-simplefile-server

I develop the Leshan-Client-Demo application to support the Firmware Update object.

To start the software download: PUT 5/0/1 coap://:/data/1_0_2.bin My 1_0_2.bin file size is 20228

The client starts download but then I receive an error data/1_0_2.bin is too large 20228 (max.: 8192)!

Questions: Is it because I get CoAP-GET and therefore the size is limited? How do I download file larger than 8192 Bytes?

Thank and Regards, Henry

For SimpleFileServer I change to public FileResource(String coapRootPath, File filesRoot) { super(coapRootPath); getAttributes().addResourceType("block"); getAttributes().setMaximumSizeEstimate(2048); this.filesRoot = filesRoot; }

For IPSO 005 FWUpdate I implement

        setState(STATE_DOWLOADING);

        //Configure the location to store the firmware.bin file
        String workingDirName = System.getProperty("user.dir");
        File packagesDir = new File(workingDirName + File.separatorChar + "packages");
        String filename = "Edison.jar";
        File packageFile = new File(packagesDir, filename);

       //Issue a CoAP-GET to the SimpleFileServer 
        CoapClient client = new CoapClient(this.packageUri);
        client.useEarlyNegotiation(512);
        CoapResponse response = client.get();
        if (response!=null) {
            System.out.println(response.getCode());
            System.out.println(response.getOptions());
            FileOutputStream out = new FileOutputStream(packageFile);
            out.write(response.getPayload());
        } else {
            System.out.println("No response received.");
        }
        setState(STATE_DOWNLOADED);
        logger.log(Level.INFO, "Download complete.");

[iot-cloud]

Hi,

I tracked out the issue with file too large

    long maxLength = NetworkConfig.getStandard().getInt(NetworkConfig.Keys.MAX_RESOURCE_BODY_SIZE);
    long length = file.length();
    if (length > maxLength) {
        LOG.warn("File {} is too large {} (max.: {})!",
                new Object[] { file.getAbsolutePath(), length, maxLength });
        exchange.respond(CoAP.ResponseCode.INTERNAL_SERVER_ERROR);
        return;
    }

However, I am now configure the MAX_RESOURCE_BODY_SIZE = 812900

I am now facing with the payload issue

    try (InputStream in = new FileInputStream(file)) {
            byte[] content = new byte[(int) length];
            int r = in.read(content);
            if (length == r) {
                Response response = new Response(CoAP.ResponseCode.CONTENT);
                response.setPayload(content);
                response.getOptions().setSize2((int) length);
                response.getOptions().setContentFormat(accept);
                exchange.respond(response);
            } else {
                LOG.warn("File {} could not be read in!", file.getAbsolutePath());
                exchange.respond(CoAP.ResponseCode.INTERNAL_SERVER_ERROR);
            }
    } catch (IOException ex) {
        LOG.warn("File {}:", file.getAbsolutePath(), ex);
        exchange.respond(CoAP.ResponseCode.INTERNAL_SERVER_ERROR);
    }

How do I go about segment my file into Block...

Thank and Regards, Henry

[iot-cloud]

All good. I think I works it out.

    try (InputStream in = new FileInputStream(file)) {
        byte[] byteArray = new byte[(int) file.length()];
        in.read(byteArray);
        exchange.respond(CONTENT, byteArray, APPLICATION_OCTET_STREAM);
    } catch (IOException ex) {
        LOG.warn("File {}:", file.getAbsolutePath(), ex);
        exchange.respond(CoAP.ResponseCode.INTERNAL_SERVER_ERROR);
    }

I believe that block size is set up using CoapClient client = new CoapClient(this.packageUri); client.useEarlyNegotiation(512); CoapResponse response = client.get();

Please correct me if I missed out anything.

Regards, Henry

[boaks]

Just to mention, this firmware-download issue is not that easy :-).

CoAP Blockwise (RFC7959) comes with many flavours :-). If PUT or GET (block1 or block2) will better work, depends on your communication setup. The PUT requires FMPOV a lot more, or simply: in my opinion that will never work for a firmware update:-)

In my experience, the device may use the GET, but also with GET there is much more to do. What should happens, if the device is not able to complete the download and must continue the download next time?

Californium's blockwise is mainly intended to overcome a situation, when sometimes the payload is larger as the UDP datagram. The file-server is intended to demonstrate the use of GET's for such firmware downloads. But a client, which will download its firmware successful in a fragile communication environment must implement his own strategies to do so. Californium just help you to disable the "transparent" blockwise. I hope, you don't plan to update a device, which can run java, to to a firmware update via coap-blockwise. And if its a c-client, its an issue for that implementation.

Today I'm too busy. I will check your comments the next days and provide you more feedback.

[sbernard31]

If you still want to use coap-blockwise, despite @boaks feedbacks/advices. You should just have to play with those parameters at both side. (earlynegociation is only needed if you want to ensure server will not use a too large blocksize)

/**
/**
 * The block size (number of bytes) to use when doing a blockwise
 * transfer. This value serves as the upper limit for block size in
 * blockwise transfers.
 */
public static final String PREFERRED_BLOCK_SIZE = "PREFERRED_BLOCK_SIZE";
/**
 * The maximum payload size (in bytes) that can be transferred in a
 * single message, i.e. without requiring a blockwise transfer.
 * 
 * NB: this value MUST be adapted to the maximum message size supported
 * by the transport layer. In particular, this value cannot exceed the
 * network's MTU if UDP is used as the transport protocol.
 */
public static final String MAX_MESSAGE_SIZE = "MAX_MESSAGE_SIZE";
/**
 * The maximum size of a resource body (in bytes) that will be accepted
 * as the payload of a POST/PUT or the response to a GET request in a
 * <em>transparent</em> blockwise transfer.
 * <p>
 * This option serves as a safeguard against excessive memory
 * consumption when many resources contain large bodies that cannot be
 * transferred in a single CoAP message. This option has no impact on
 * *manually* managed blockwise transfers in which the blocks are
 * handled individually.
 * <p>
 * Note that this option does not prevent local clients or resource
 * implementations from sending large bodies as part of a request or
 * response to a peer.
 * <p>
 * The default value of this property is
 * {@link NetworkConfigDefaults#DEFAULT_MAX_RESOURCE_BODY_SIZE}.
 * <p>
 * A value of {@code 0} turns off transparent handling of blockwise
 * transfers altogether.
 */
public static final String MAX_RESOURCE_BODY_SIZE = "MAX_RESOURCE_BODY_SIZE";

I hope, you don't plan to update a device, which can run java, to to a firmware update via coap-blockwise

I believe that @boaks would say that if you are able to run java on your device, you should easily download your file using a protocol adapted to download large file (as http or ftp ...)

[boaks]

@boaks feedback.

GET works, but in the field not easy out of the box :-).

I will update the setting of the file-server. It's long ago, when I introduced it. But the clients must consider there communication environment. Tha's not out of the box.

[boaks]

I prepared a PR #793, which prepares the properties to do a file download up to 2MB. That client is just useful to demonstrate it, it is not something which could be used in a real world communication. The basic change you need, is to adjust the MAX_RESOURCE_BODY_SIZE to your expected file size on both sides, server and client. You may just edit the "Californium.properties". In the future, the update will do that, if you delete the Californium.properties of the directory you run the fileserver and get-cleint demos and the updated demos create new "Californium.properties".

You may fetch the PR either from eclipse using git fetch origin pull/ID/head:BRANCHNAME

or the update_fileserver branch from

https://github.com/bsinno/californium/tree/update_fileserver