search

eclipse-leshan / leshan

Java Library for LWM2M
https://www.eclipse.org/leshan/
BSD 3-Clause "New" or "Revised" License
653 stars 408 forks source link

Question - How we check compliance Leshan with standard #1115

Open Michal-Wadowski opened 3 years ago

Michal-Wadowski commented 3 years ago

How are we sure that Leshan project is properly implemented with standards? Do you execute some periodically external tests or compliance is checked only while code reviewing?

sbernard31 commented 3 years ago

How are we sure that Leshan project is properly implemented with standards?

When you talk about "standards", you mean LWM2M or any standard that Leshan is based on ?

Do you execute some periodically external tests or compliance is checked only while code reviewing?

I don't know about this kind of external tests. Currently we rely on code review + our unit tests + feedback of our community.

Michal-Wadowski commented 3 years ago

Sorry, I didn't crystallize what I mean :smile: Of course I mean LWM2M :smile:

Currently we rely on code review + our unit tests + feedback of our community.

Ok, we currently study of possibility for compliance testing with a third party. But only once because it costs :smile:

sbernard31 commented 3 years ago

Ok, we currently study of possibility for compliance testing with a third party.

That could be great. :+1: Could you share who does this kind of compliance test ?

But only once because it costs smile

If this is a one shot maybe we should keep it for a the final RC release, or ?

Michal-Wadowski commented 3 years ago

For some reasons we have to perform tests sooner rather than later...

Nevertheless, maybe you have some preferences regarding tests? The potential options that we can perform are:

int-0   Client Initiated Bootstrap
int-1   Client Initiated Bootstrap Full (PSK)
int-2   Client Initiated Bootstrap Full (Cert)
int-3   Simple Bootstrap from Smartcard
int-4   Bootstrap Delete
int-5   Server Initiated Bootstrap
int-6   Bootstrap Sequence
int-7   Fallback to bootstrap
int-8   Bootstrap Read
int-9   Bootstrap and Configuration Consistency

int-101 Initial Registration
int-102 Registration Update
int-103 Deregistration
int-104 Registration Update Trigger
int-105 Discarded Register Update
int-106 TCP Binding
int-107 Extending the lifetime of a registration
int-108 Turn on Queue Mode
int-109 Behavior in Queue Mode

int-201 Querying basic information in Plain Text format
int-203 Querying basic information in TLV format
int-204 Querying basic information in JSON format
int-205 Setting basic information in Plain Text format
int-211 Querying basic information in CBOR format
int-212 Setting basic information in CBOR format
int-215 Setting basic information in TLV format
int-220 Setting basic information in JSON format
int-221 Attempt to perform operations on Security Object (ID 0)
int-222 Read on Object
int-223 Read on Object Instance
int-224 Read on Resource
int-225 Read on Resource Instance
int-226 Write (Partial Update) on Object Instance
int-227 Write (replace) on Resource
int-228 Write on Resource Instance
int-229 Read-Composite Operation
int-230 Write-Composite Operation
int-231 Querying basic information in SenML JSON format
int-232 Querying basic information in SenML CBOR format
int-233 Setting basic information in SenML CBOR format
int-234 Setting basic information in SenML JSON format
int-235 Read-Composite Operation on root path
int-236 Read-Composite Partial Presence
int-237 Read on Object without specifying Content-Type
int-241 ExecuResource Rebooting the device
int-256 Write Operation Failure
int-257 Write-Composite Operation
int-260 Discover Command
int-261 Write-Attribute Operation on a multiple resource
int-280 Successful Read-Composite Operation
int-281 Partially Successful Read-Composite Operation

int-301 Observation and Notification of parameter values
int-302 Cancel Observations using Reset Operation
int-303 Cancel observations using Observe with Cancel parameter
int-304 Observe-Composite Operation
int-305 Cancel Observation-Composite Operation .
int-306 Send Operation
int-307 Muting Send
int-308 Observe-Composite and Creating Object Instance
int-309 Observe-Composite and Deleting Object Instance
int-310 Observe-Composite and modification of parameter values
int-311 Send command

int-401 UDP Channel Security Pre-shared Key Mode
int-402 UDP Channel Security Certificate Mode .
int-403 UDP Channel Security Certificate Mode (Server Identify Verification Failure) .
int-404 TCP Channel Security Certificate Mode ..
int-405 OSCORE Security

int-651 Check Access to the Resources
int-652 Querying the firmware version from the client
int-680 Create Object Instance
int-685 Delete Object Instance
int-701 Querying the readable resources of object
int-710 Basic Observation and notification on Connectivity Monitoring Object Resources
int-751 Querying the readable resources
int-755 Setting the wriResource Package
int-756 Setting the wriResource Package URI
int-760 Basic Observation and notification on Firmware Update Object Resources
int-770 Successful Firmware update (via COAP)
int-771 Successful Firmware update (via alternative mechanism)
int-772 Error Case 1 firmware installation without downloaded package
int-773 Error Case 2 shortage of storage memory
int-774 Error Case 3 out of memory
int-775 Error Case 4 Connection lost during download (package URI)
int-776 Error Case 5 Package Integrity check failure
int-777 Error Case 6 unsupported package type
int-778 Error Case 7 invalid URI (package URI)
int-779 Error Case 8 Unsuccessful Firmware Update
int-801 Querying the readable resources of object
int-810 Basic Observation and notification on Location Object Instance
int-901 Querying a Data Collection from Connectivity Object Instance
int-905 Setting the wriresources
int-910 Basic Observation and notification on Connectivity Monitoring Object Instance
int-950 Multi-Servers Registration
int-951 Multi-Servers & Attributes
int-1200    Querying the readable resources of Object ID:10
int-1201    Querying the readable resources of Object ID:10 in version 1.1
int-1202    Setting the Power Saving Mode Resources of Object ID:10 (version 1.1 only)
int-1203    Observation and notification on Object ID:10 related to Power Saving Mode Resources (version 1.1 only)
int-1204    Observation and notification on Object ID 10 related to timers
int-1250    APN configuration
int-1350    Bearer Selection
int-1630    Create Portfolio Object Instance
int-1635    Delete Portfolio Object Instance
int-1900    Observation and notification on Object ID 19
int-1901    Setting the wriResource Data
int-2000    Control the Log Function
int-2001    Querying the readable resources

What do you think, some tests would be more preferable than other? Or some tests should not be performed because lack of implementation?

I think the tests are based on: http://www.openmobilealliance.org/release/lightweightm2m/ETS/OMA-ETS-LightweightM2M-V1_1-20190912-D.pdf

sbernard31 commented 3 years ago

Tests are for clients, servers or bootstrap servers ?

What do you think, some tests would be more preferable than other?

I don't know :thinking: ... I guess the most we tests the better it is :sweat_smile: ? maybe the cost depends of the number of tests ?

Or some tests should not be performed because lack of implementation?

I can have a first look and try to identify tests which will not pass.

Michal-Wadowski commented 3 years ago

Tests are for clients, servers or bootstrap servers ?

We are focusing on server side. The client code is by the way.

maybe the cost depends of the number of tests ?

That's right. But we don't specified yet how many tests we will perform. We still debate about that.

I can have a first look and try to identify tests which will not pass.

Thx! It will help us a lot :smiley:

sbernard31 commented 3 years ago

A general remarks, I don't really know what you will tests ? I mean demos or library ?

Demos works pretty much out of the box but does not support everything supported by the library. Library supports much more but that means you need to code a runnable program with it to make it works ? Or maybe you test your product which integrates Leshan ?

Michal-Wadowski commented 3 years ago

Good point. Leshan is just a library and without proper usage it can't be tested enough. I think the demos might be not enough to perform all tests. We have to think about that.

sbernard31 commented 3 years ago
test comment
int-3 Simple Bootstrap from Smartcard I think this mainly concerns client side but by default we don't really support smartcard even if I'm not sure your what it could means to support it in the library
int-106 TCP Binding not yet implemented, I'm not even sure we have a coap+tcp mature java implementation available
int-235 Read-Composite Operation on root path Read Composite is implemented but not about targeting root path
int-260 Discover Command write attributes not implemented at client side
int-261 Write-Attribute Operation on a multiple resource write attributes not implemented at client side
int-301 Observation and Notification of parameter values write attributes not implemented at client side
int-307 Muting Send mute is not implemented at client side but should be pretty easy
int-404 TCP Channel Security Certificate Mode .. TCP not implemented
int-405 OSCORE Security in development but not manage by me, so not so sure of the current state. Currently not in master
from 651 to 910 this is about testing object implementation but currently client library is more about letting you implement your object
int-950 Multi-Servers Registration multi server at client side is not implemented
int-951 Multi-Servers & Attributes multi server at client side is not implemented
from 1200 to 2001 this is about testing object implementation but currently client library is more about letting you implement your object
sbernard31 commented 3 years ago

I did a really quick look. The tests not mentioned above :point_up: should be OK

sbernard31 commented 2 years ago

I think this question is maybe a bit related to : https://github.com/eclipse/wakaama/issues/617

sbernard31 commented 2 years ago

(Just some thought about this, maybe a security audit about Leshan would be a better investment rather than a LWM2M specification compliance one)

sbernard31 commented 1 year ago

It seems that most of http://www.openmobilealliance.org/release/lightweightm2m/ETS/OMA-ETS-LightweightM2M-V1_1-20190912-D.pdf pass with Zephir, see : https://github.com/zephyrproject-rtos/zephyr/pull/64013