search

yamcs / yamcs-scos2k

0 stars 0 forks source link

Load data #1

Open ClaudMor opened 5 years ago

ClaudMor commented 5 years ago

Hello,

I'm absolutely new to Yamcs. From the notes it is clear how to load the SCOS2K tables. But how does one load the actual ccsds data?

Thanks in advance

xpromache commented 5 years ago

Hello, can you please describe what you want to achieve?

ClaudMor commented 5 years ago

Hello,

Thanks for your time.

I have ccsds data and relative SCOS2K loaders. How may I inject them into yamcs, so that the ccsds data are interpreted and the result gets passed to the HTTP server, just as the simulator + yamcs-openmct-plugin work?

xpromache commented 5 years ago

did you manage to install and run yamcs at all?

I suggest you first try to load the file in the packet-viewer bin/packet-viewer.sh - this should work without having the server started. You need first to make the mdb.yaml as described in the readme.

BTW: if you can share your MIB and test files, I can add some test cases on the public site. The MIB I'm using for testing cannot be made public.

ClaudMor commented 5 years ago

Yes, we managed to start yamcs using "service yamcs-server start". We installed the scos2k plugin, the simulator and the yamcs-openmct plugin. The simulator and the yamcs-openmct plugin actually work as expected.

Unfortunately, we cannot share the MIB and test files.

Does the mdb.yaml file need to be in the etc folder of the server? Does any documentation regarding the use of the .sh scripts exist?

Thanks again

xpromache commented 5 years ago

If you installed the server there is a file /opt/yamcs/etc/mdb.yaml on the server. You can edit that one to have the mdb defined.

Then you can start the /opt/yamcs/bin/packet-viewer.sh and do File->Open and in the XTCE DB on top, you should see the name of the config you added in mdb.yaml. Then you can select the CCSDS file and see if it loads. This will allow to verify that yamcs can decode properly the packets according to your MIB.

I must say that we (at SpaceApps) have never worked with the openmct and I don't know what it is doing. Do you want to load the file in the yamcs archive or somehow "play" it to simulate data coming in realtime? I don't know if openmct is able to access the Yamcs archive.

ClaudMor commented 5 years ago

If i put the mdb.yaml file in /opt/yamcs/etc on the server and then start the server and the packet-viewer, the latter tells me that mdb.yaml is missing. But if i put the mdb.yaml in /opt/yamcs-client/etc, then it shows an error window stating "Cannot find a mapping for key ..."

xpromache commented 5 years ago

Check the permission of the mdb.yaml, it should be readable by the user that executed packet-viewer.sh I just tried installing the yamcs 4.9.5 rpm, made a mdb.yaml and then I can start the packet viewer and load a file.

For the standalone packet-viewer, it is not necessary to run the server. After we get this running, we will be able to connect it to the server but first let's try in standalone mode.

xpromache commented 5 years ago

If i put the mdb.yaml file in /opt/yamcs/etc on the server and then start the server and the packet-viewer, the latter tells me that mdb.yaml is missing. But if i put the mdb.yaml in /opt/yamcs-client/etc, then it shows an error window stating "Cannot find a mapping for key ..."

Sorry, now I realize that you start packet-viewer from /opt/yamcs-client, then yes you copy the mdb.yaml in the etc in that directory. Can you paste it here to see what's wrong?

ClaudMor commented 5 years ago

I set permissions for all users but i keeps giving the same error. Anyway i was executing packet-viewer as root. So the command i'm giving is ./packet-viewer.sh -x scos-mib [telemetry-file-absolute-path]. The [] are of course excluded.

xpromache commented 5 years ago

I set permissions for all users but i keeps giving the same error. Anyway i was executing packet-viewer as root. So the command i'm giving is ./packet-viewer.sh -x scos-mib [telemetry-file-absolute-path]. The [] are of course excluded.

can you show me your mdb.yaml?

ClaudMor commented 5 years ago

the mdb.yaml is:

scos-mib:

where path/to/ASCII is replaced with the path to the mib github replaces the "-" of yaml with a dot

xpromache commented 5 years ago

I see, it's exactly the example from the readme file; can you then write the complete error?

ClaudMor commented 5 years ago

No columns definition found in ui preferences May 28 17:05:29.627 [1] org.yamcs.ui.packetviewer.PacketViewer loadLocalXtcedb SEVERE:cannot find a mapping for key 'TM': cannot find a mapping for key 'TM' org.yamcs.YConfiguration.checkKey(YConfiguration.java:261) org.yamcs.YConfiguration.getMap(YConfiguration.java:340) org.yamcs.scos2k.TmMibLoader.(TmMibLoader.java:102) org.yamcs.scos2k.TcMibLoader.(TcMibLoader.java:79) org.yamcs.scos2k.MibLoader.(MibLoader.java:23) sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) java.lang.reflect.Constructor.newInstance(Constructor.java:423) org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78) org.yamcs.xtceproc.XtceDbFactory.getLoaderTree(XtceDbFactory.java:510) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:104) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:79) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:67) org.yamcs.ui.packetviewer.PacketViewer.loadLocalXtcedb(PacketViewer.java:518) org.yamcs.ui.packetviewer.PacketViewer.openFile(PacketViewer.java:488) org.yamcs.ui.packetviewer.PacketViewer.main(PacketViewer.java:1173)

xpromache commented 5 years ago

ah, sorry, the README file was not up2date. I just updated it now.

ClaudMor commented 5 years ago

now it gives me:

No columns definition found in ui preferences (error line that reports an "unknown" parameter type. Unfortunately these lines also report confidential information so i can't paste them here) org.yamcs.scos2k.MibLoaderBits.getDataEncoding(MibLoaderBits.java:211) org.yamcs.scos2k.TmMibLoader.createParameterTypeNcateg(TmMibLoader.java:517) org.yamcs.scos2k.TmMibLoader.createParameterType(TmMibLoader.java:490) org.yamcs.scos2k.TmMibLoader.loadPfc(TmMibLoader.java:361) org.yamcs.scos2k.TmMibLoader.loadTelemetryParams(TmMibLoader.java:440) org.yamcs.scos2k.TmMibLoader.loadTelemetry(TmMibLoader.java:113) org.yamcs.scos2k.MibLoader.load(MibLoader.java:30) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:720) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:723) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:145) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:79) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:67) org.yamcs.ui.packetviewer.PacketViewer.loadLocalXtcedb(PacketViewer.java:518) org.yamcs.ui.packetviewer.PacketViewer.openFile(PacketViewer.java:488) org.yamcs.ui.packetviewer.PacketViewer.main(PacketViewer.java:1173)

What could it be?

xpromache commented 5 years ago

I see at that line the exception is "Unknonw parameter type (ptc,pfc)". But what are your parameter ptc,pfc (these should be just some numbers)?

ClaudMor commented 5 years ago

Yes, we had a non-standard ptc,pfc . Probably it hasn't been implemented, we'll just remove it. Thank so much, we'll let you know how it goes!

ClaudMor commented 5 years ago

Hello again,

We have another blocker issue: our MIB uses the symbol "<>" to mean the "different" symbol ("!="). Th problem is that the packet-viewer generates a java class from the file containing "<>" without translating "<>" into "!=", but it keeps the "<>" as it is in the generated java class. This causes a compilation error, because java does not recognize the symbol "<>" .

How could this be solved?

Thanks for your support

xpromache commented 5 years ago

Can you please edit yamcs-scos2k/src/main/java/org/yamcs/scos2k/ol/Operation.java and change "<>" into "!=" on line 37: final static Operation DIFFER = new Operation("<>", COMP);

And then recompile (mvn install) and copy the jar into the yamcs lib.

ClaudMor commented 5 years ago

Thanks, it worked.

we would like to signal to you that we noticed also that : OCF_CODIN=I and CDF_ELTYPE=F are not supported.

How could this be solved?

Moreover, after implementing your solution about the "<>", we were given an error about a null pointer exception, that we solved by changing name -> parameters.get(name).ptype in name->parameters.get(mp.name).ptype in file TmMibLoader.java (line 402) (was it a good move, or did we just cause a subtler chain of errors?)

thanks for your realtime support!!

xpromache commented 5 years ago

  1. OCF_CODIN=I is not supported Looking in the code I see it's supposed to work for numeric parameters but apparently yours is not numeric (most likely there is a bug somewhere else). Can you show the exact error?

  2. CDF_ELTYPE=F I understand this means that the command argument is not editable. I'm not sure what the difference between this and the fixed area (CDF_ELTYPE=A) is. In any case, from what I can see, XTCE does not support read-only command arguments as such. What we could do is to create an abstract command and the real command can be inheriting from the abstract one and then it's possible to fix the arguments of the parent command.

Moreover, after implementing your solution about the "<>", we were given an error about a null pointer exception, that we solved by changing name -> parameters.get(name).ptype in name->parameters.get(mp.name).ptype in file TmMibLoader.java (line 402) (was it a good move, or did we just cause a subtler chain of errors?)

I'm not sure your modification is good. The null pointer exception you get I think it's because the parameter that goes into the formula is not known. We should do better checking and error reporting in this case.

ClaudMor commented 5 years ago
  1. This is the error stack for the OCF_CODIN=I (that we anyway "solved" by modifying the MIBs) `OCF_CODIN = I cannot be used for a parameter of type class org.yamcs.xtce.EnumeratedParameterType org.yamcs.scos2k.TmMibLoader.loadLimits(TmMibLoader.java:699) org.yamcs.scos2k.TmMibLoader.loadTelemetry(TmMibLoader.java:116) org.yamcs.scos2k.MibLoader.load(MibLoader.java:30) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:720) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:723) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:145) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:79) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:67) org.yamcs.ui.packetviewer.PacketViewer.loadLocalXtcedb(PacketViewer.java:518) org.yamcs.ui.packetviewer.PacketViewer.openFile(PacketViewer.java:488) org.yamcs.ui.packetviewer.PacketViewer.main(PacketViewer.java:1173)

`

  1. unfortunately we may not share the MIBs, but we noticed that one can solve the problem by again modifying the MIBs

  2. Was not able to reproduce the error after correcting the other ones, probably it was a consequence of other mistakes.

Anyway now it seems to work (parameters appear in the packet-viewer window).

Next step would be to have something like the simulator: so i think we should pass the data to the server and create an instance. Could you please help us? How to do it? It seems i can't find exhaustive instructions in the documentation.

xpromache commented 5 years ago

What you can do now is to modify the yamcs.simulator.yaml and add this under the data links:

dataLinks:
  - name: file_import
    class: org.yamcs.tctm.FilePollingTmDataLink
    args:
       stream: tm_realtime
       incomingDir: "/path/to/incoming/data/dir"
       deleteAfterImport: false
       delayBetweenPackets: 1000

This will create you a data link called "file_import" with the data replayed from a file in the directory you specify (/path/to/incoming/dir). Make sure that directory has no other file inside than the one you have to replay. You can remove the other links or set enabledAtStartup: false to make sure they are not generating bogus data. Then you start or restart yamcs server (service yamcsd restart) and I suggest first connect with the web browser (http://yamcs-server-host:8090/ and go to the Links on the left to see that the data_in counter is increasing. Also check the logs for errors /opt/yamcs/log/yamcs-server.log.0

xpromache commented 5 years ago
  1. This is the error stack for the OCF_CODIN=I (that we anyway "solved" by modifying the MIBs) `OCF_CODIN = I cannot be used for a parameter of type class org.yamcs.xtce.EnumeratedParameterType

I guess you have the OCF_INTER=U - that means the limit check has to be performed on the raw value. Currently this is ignored (all limits are performed on the engineering value) - so this is a bug in the loader.

I must say that the SCOS2k loader has been made as a proof of concept and used only on a limited MIB (which as we can see, was not using all features). If you decide to actually use it, I will go and fix those bugs. So please let me know...

ClaudMor commented 5 years ago

Hello,

Yes, we will tell you more after the week-end! thank you for the moment, it seems to be working!

ClaudMor commented 5 years ago

Hello,

When i said Anyway now it seems to work (parameters appear in the packet-viewer window). I was actually wrong: the packet viewer showed a few packets, but not all of them. So we exported the data as hexdump and pointed the packet-viewer to the exported file. Now the packet viewer seems to load all packets, but it gives the error:

No columns definition found in ui preferences Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Object.clone(Native Method) at org.yamcs.ui.packetviewer.ListPacket.<init>(ListPacket.java:31) at org.yamcs.ui.packetviewer.PacketsTable.packetReceived(PacketsTable.java:507) at org.yamcs.ui.packetviewer.PacketViewer$3.process(PacketViewer.java:701) at javax.swing.SwingWorker$3.run(SwingWorker.java:414) at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112) at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.run(SwingWorker.java:832) at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112) at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.actionPerformed(SwingWorker.java:842) at javax.swing.Timer.fireActionPerformed(Timer.java:313) at javax.swing.Timer$DoPostEvent.run(Timer.java:245) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82) Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space

and then, when i click on a packet on the left tree, some warnings about corrupted files and SEQ_COUNT_JUMP appear.

Anyway we tried to start the server after modifying the dataLinks as you told us and the mdb.yaml in the server according to the readme. The file_import link is created and active, but the In_Count is zero. It keeps creating new server.out.log.0 files in /opt/yamcs/log, all of them just repeating the following error over and over:

Jun 03 18:12:17.136 [19] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /data/file Jun 03 18:12:17.136 [19] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /data/file: : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (exadecimalString) org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118) org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84) com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) com.google.common.util.concurrent.Callables$3.run(Callables.java:100) java.lang.Thread.run(Thread.java:748)

Where /data/file is the path to the data file. The exadecimalString is a exadecimal string longer than a packet header. It infact is a frame header. It seems that a packet header is expected. Is there an already implemeted way to extract packets from frames?

Thank you very much

xpromache commented 5 years ago

Is your file containing packets or frames? What kind of frames?

Both packet viewer and FilePollingTmDataLink will only work with binary data, not hexadecimal.

ClaudMor commented 5 years ago

Ok so the input needs to be a binary and NOT textual file. Our file contains frames with packets inside. If knowing the kind of frame is crucial to proceed, we will be able to tell you in 2-3 days at worst. If the the file must contain packets and not frames, is there an already-implemented way to extract packets from frames? (It gives me very similar errors in the server.logs if i point the server to the binary data and not the textual ones)

xpromache commented 5 years ago

I don't know where you got the data from but I suspect it's either AOS frames: https://public.ccsds.org/Pubs/732x0b3e1.pdf or TM frames: https://public.ccsds.org/Pubs/132x0b2.pdf

The AOS frames start with first two bits 01 and the TM frame with 00. So you can probably tell from that.

In both cases, what you need to know is the frame size. Without it I'm afraid you cannot split the file in frames. But as you can see in the pdfs above, the first 10/12 bits is what they called Master Channel ID and that repeats in each frame. So you can try to guess the frame length from there.

ClaudMor commented 5 years ago

Ok so thanks to your advice we discovered the kind of frame, and where to eventually split that. But we saw something strange: if we rely on the simulator, shouldn't the content of your readme go in /etc/yamcs.simulator.yaml, under the mdb section? (rather than in mdb.yaml?) I ask this question because i noticed that if i delete all the mdb loaders in etc/yamcs.simulator.yaml and replace them with your readme content, while also keeping the mdb.yaml as before (so with your readme content copied inside), then intentionally-generated typos in the mdb.yaml are not reported as errors, while typos in the (for example) string "org.yamcs.scos2k.MibLoader" in yamcs.simulator.yaml are. Moreover: if i do delete all the mdb loaders in etc/yamcs.simulator.yaml and replace them with your readme content, even without misspelling, it keeps telling me: 7:59:20.101 [1] YamcsServer yamcs 4.9.5, build 8424c81d32e33aa1ba804f913e2bf4bbe4cc588d 17:59:20.267 [1] YamcsServer Loading global service org.yamcs.web.HttpServer 17:59:20.454 [1] YamcsServer Loading global service org.yamcs.tse.TseCommander 17:59:20.473 [1] YamcsServer Loading online instance 'simulator' 17:59:20.488 [1] YamcsServer SEVERE Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284) org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: Cannot instantiate object from class org.yamcs.scos2k.MibLoader: java.lang.ClassNotFoundException: org.yamcs.scos2k.MibLoader org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:95) org.yamcs.xtceproc.XtceDbFactory.getLoaderTree(XtceDbFactory.java:510) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:104) org.yamcs.xtceproc.XtceDbFactory.getInstance(XtceDbFactory.java:620) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:98) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: java.lang.ClassNotFoundException: org.yamcs.scos2k.MibLoader java.net.URLClassLoader.findClass(URLClassLoader.java:382) java.lang.ClassLoader.loadClass(ClassLoader.java:424) sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) java.lang.ClassLoader.loadClass(ClassLoader.java:357) java.lang.Class.forName0(Native Method) java.lang.Class.forName(Class.java:264) org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:30) org.yamcs.xtceproc.XtceDbFactory.getLoaderTree(XtceDbFactory.java:510) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:104) org.yamcs.xtceproc.XtceDbFactory.getInstance(XtceDbFactory.java:620) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:98) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762)

while the yamcs.simulator.yaml is:

`services:

dataLinks:

mdb:

streamConfig: tm:

Thanks again, hope to able to give you more precise answers.

xpromache commented 5 years ago

sorry for the confusion. Basically in yamcs.simulator.yaml you either put an entire config like you did, or you put 

mdb: key-from-mdb-yaml

where key-from-mdb-yaml is the key of a config from mdb.yaml (e.g. scos-mib)

The only thing is that the packet viewer can work only with an mdb defined in the mdb.yaml, not in the yamcs.instance.yaml which is a config read by the yamcs server only.

The error you get is related to not finding the org.yamcs.scos2k.MibLoader which means that probably the yamcs-scos2k jar is not in the lib or lib/ext directory of the yamcs server (/opt/yamcs/lib/ normally). Or maybe the jar is not readable by the user that runs yamcs.

ClaudMor commented 5 years ago

Thanks for solving the "org.yamcs.scos2k.MibLoader not found" exception! So if we just leave

mdb: scos-mib

in yamcs.simulator.yaml, it throws us:

Jun 06 10:14:51.355 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
    org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
    org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: Unknown sequence container: /YSS/SIMULATOR/tm2_container
    org.yamcs.StreamConfig.<init>(StreamConfig.java:80)
    org.yamcs.StreamConfig.lambda$getInstance$0(StreamConfig.java:34)
    java.util.HashMap.computeIfAbsent(HashMap.java:1127)
    org.yamcs.StreamConfig.getInstance(StreamConfig.java:34)
    org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:51)
    org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:39)
    org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99)
    org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)

Jun 06 10:14:51.356 [11] org.yamcs.yarch.rocksdb.RDBFactory shutdown FINE:shutting down, closing 1 databases under /storage/yamcs-data/simulator.rdb: []

while if i leave the other MIBs,

mdb:
  # Configuration of the active loaders
  # Valid loaders are: sheet, xtce or fully qualified name of the class
  - type: "sheet"
    spec: "mdb/simulator-ccsds.xls"
    subLoaders:
      - type: "sheet"
        spec: "mdb/landing.xls"
      - type: "sheet"
        spec: "mdb/GS-tmtc.xls"
  - type: "org.yamcs.xtce.TseLoader"
    subLoaders:
      - type: "sheet"
        spec: "mdb/tse/simulator.xls"
  #Loads the performance testing mission database
  - type: "org.yamcs.simulator.PerfMdbLoader"
    args:
      numPackets: 18
      packetSize: 1476
  - type: "org.yamcs.scos2k.MibLoader" 
    args: 
        path: "/path/to/ASCII"        
        epoch: "UNIX"
        #TC specific settings
        TC:   
           # default size in bytes of the size tag for variable length strings and bytestrings
           vblParamLengthBytes: 0
        #TM specific settings
        TM:  
            vblParamLengthBytes: 1
            # byte offset from beginning of the packet where the type and subType are read from     
            typeOffset: 7
            subTypeOffset: 8

it throws:

Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /path/to/data
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /path/to/data : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (unreadable string)
    org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118)
    org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84)
    com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
    com.google.common.util.concurrent.Callables$3.run(Callables.java:100)
    java.lang.Thread.run(Thread.java:748)

over and over. So the solution is to keep the second configuration and split the data in frames, right?

xpromache commented 5 years ago

The reason for this error is that in the streamConfig section you have 

 tm:
    ...
    - name: "tm2_realtime"
      rootContainer: "/YSS/SIMULATOR/tm2_container"

which means that data that comes on the tm2_realtime stream has to be processed according to the tm2_container. This container doesn't exist since you use your own MDB.

So, just remove that "rootContainer" - meaning that the tm2_realtime data will be processed according to the default root container from your MDB. In reality you could remove the tm2_realtime all together but you have to also remove it in the data link section.

FYI: in Yamcs a Datalink puts data into a Stream from where it is processed by a Processor or recorded in the so called stream database (based on rocksdb).

ClaudMor commented 5 years ago

Thanks, it worked.

Now , using

mdb: scos-mib

it gives the second error:

Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /path/to/data
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /path/to/data : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (unreadable string)
    org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118)
    org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84)
    com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
    com.google.common.util.concurrent.Callables$3.run(Callables.java:100)
    java.lang.Thread.run(Thread.java:748)
xpromache commented 5 years ago

ok, that's because the file you put in there is ASCII and it thinks it's another format (Pacts - some old EGSE software, we have in fact to remove it from the open-source yamcs).

I'm afraid you have now to sort out the data format problem, and I see two options:

  1. you extract packets out of frames into a different file and put that file to be read by yamcs.

  2. you use the yamcs framing support but unfortunately we cannot read frames from files at this moment. What you can do is to make a small python script to send the frames one by one via UDP. In the simulator configuration, yamcs expects frames via UDP on port 10017 Make sure you tune the parameters

    args:
   port: 10017 #UDP port to listen to
   #one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
   frameType: "AOS"
   spacecraftId: 0xAB
   frameLength: 512
   frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames
   insertZoneLength: 0
   errorCorrection: CRC16 #NONE, CRC16 or CRC32 (only for USLP)

    according to your data.

ClaudMor commented 5 years ago

Ok. Which format should the frames be sent in? An array of bytes?

xpromache commented 5 years ago

One UDP packet = one frame. An array of bytes if you want but binary please no hexadecimal or any formatting.

On Thu, Jun 6, 2019 at 3:14 PM claudio20497 notifications@github.com wrote:

Ok. Which format should the frames be sent in? An array of bytes?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR35TP4AZKTBV36G7UTPZEEUHA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXCZM3Q#issuecomment-499488366, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR5F7AVBPY3VOQWYJZTPZEEUHANCNFSM4HP2SDXA .

ClaudMor commented 5 years ago

Ok. So we no longer use the dataLink FilePollingTmDataLink, right?

xpromache commented 5 years ago

That's right. That one only works with packets.

On Thu, Jun 6, 2019 at 3:43 PM claudio20497 notifications@github.com wrote:

Ok. So we no longer use the dataLink FilePollingTmDataLink, right?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR4GBOR5W5JB6U5QTU3PZEH65A5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXC4AYY#issuecomment-499499107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR26HFKVN4LFZWFG4UTPZEH65ANCNFSM4HP2SDXA .

ClaudMor commented 5 years ago

Hello,

So now data arrives to the simulator, but it fails to send them to the server. We have frame containing ccsds data. we use, as packetpreprocessor, packetPreprocessorClassName: org.yamcs.tctm.CcsdsPacketInputStream

with no arguments, as the documentation (https://www.yamcs.org/yamcs/javadoc/org/yamcs/tctm/CcsdsPacketInputStream.html) seems to suggest. Is it correct? Looking at the error below, it appears to need an argument.

Jun 06 19:07:33.012 [1] org.yamcs.tctm.ccsds.VirtualChannelPacketHandler[simulator] <init> SEVERE:Cannot instantiate the packet preprocessor: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
    org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:75)
    org.yamcs.tctm.ccsds.VirtualChannelPacketHandler.<init>(VirtualChannelPacketHandler.java:56)
    org.yamcs.tctm.ccsds.TmManagedParameters.createVcHandlers(TmManagedParameters.java:65)
    org.yamcs.tctm.ccsds.MasterChannelFrameHandler.<init>(MasterChannelFrameHandler.java:52)
    org.yamcs.tctm.UdpTmFrameLink.<init>(UdpTmFrameLink.java:57)
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
    org.yamcs.tctm.DataLinkInitialiser.createDataLink(DataLinkInitialiser.java:98)
    org.yamcs.tctm.DataLinkInitialiser.<init>(DataLinkInitialiser.java:57)
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
    org.yamcs.YamcsServer.createService(YamcsServer.java:705)
    org.yamcs.YamcsServer.createServices(YamcsServer.java:145)
    org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:101)
    org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)

Jun 06 19:07:33.012 [1] org.yamcs.YamcsServer createServices SEVERE:Cannot create service org.yamcs.tctm.DataLinkInitialiser, with arguments null: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
Jun 06 19:07:33.013 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
    org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
    org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
    org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:75)
    org.yamcs.tctm.ccsds.VirtualChannelPacketHandler.<init>(VirtualChannelPacketHandler.java:56)
    org.yamcs.tctm.ccsds.TmManagedParameters.createVcHandlers(TmManagedParameters.java:65)
    org.yamcs.tctm.ccsds.MasterChannelFrameHandler.<init>(MasterChannelFrameHandler.java:52)
    org.yamcs.tctm.UdpTmFrameLink.<init>(UdpTmFrameLink.java:57)
    sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
    org.yamcs.tctm.DataLinkInitialiser.createDataLink(DataLinkInitialiser.java:98)
    org.yamcs.tctm.DataLinkInitialiser.<init>(DataLinkInitialiser.java:57)

Moreover, do you think that this is the right preprocessor to read ccsds frame and process ccsds packet to be sent to the yamcs server?

Thanks a lot

xpromache commented 5 years ago

that CccsdsPacketInputStream is splitting a stream into packets. That doesn't work with frames (it works if you have a file of packets or if you send packets via a tcp connection).

What I suggested is that you use the config from the simulator that receives frames via UDP. Then you don't need to split any stream because UDP is providing the splitting.

If you look here: https://github.com/yamcs/yamcs/blob/master/yamcs-simulation/etc/yamcs.simulator.yaml you see that the tm2_realtime and tm_realtime use the PacketInputStream (although it is not explicitley configured for tm_realtime).

The UDP_FRAME_IN link uses UDP (so no stream splitter) and provides the functionality for extracting the packets from the frames. I think this is the one you want but you need to send somehow the frames to yamcs. Note that the simulator has nothing to do in your setup and I suggest you remove it from the config file (remove the section org.yamcs.simulation.simulator.SimulatorCommander) because it generates packets and frames which you don't want.

ClaudMor commented 5 years ago

Ok thanks.

So i noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not. If the packets were sent to and received by those vcs, then they would be put on the configurated streams and passed to the server (right?). if it is right,
at the moment it may not happen because the configuration of those is does not reflect our data (i think). Is there a way to know what the other possible config values are?

EDIT: if i go to monitor->events (in the web page), i see a lot of Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number) under the "message" column, from type and source UdpTmFrameLink.

ClaudMor commented 5 years ago

After doing some modifications, we stumbled upon this error:

Jun 07 18:14:18.276 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
    org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
    org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: java.lang.NullPointerException
    org.yamcs.yarch.rocksdb.RdbPartitionManager.addPartitionByTimeAndValue(RdbPartitionManager.java:110)
    org.yamcs.yarch.rocksdb.RdbPartitionManager.readPartitions(RdbPartitionManager.java:47)
    org.yamcs.yarch.rocksdb.RdbStorageEngine.loadTable(RdbStorageEngine.java:84)
    org.yamcs.yarch.YarchDatabaseInstance.loadTables(YarchDatabaseInstance.java:148)
    org.yamcs.yarch.YarchDatabaseInstance.<init>(YarchDatabaseInstance.java:96)
    org.yamcs.yarch.YarchDatabase.getInstance(YarchDatabase.java:75)
    org.yamcs.StreamInitializer.<init>(StreamInitializer.java:46)
    org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:37)
    org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99)
    org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
    org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
    org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
    org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
    org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
    org.yamcs.YamcsServer.main(YamcsServer.java:762)

which throws no matter the config we use. Anyway our yamcs.sumulator.yaml is:

services:
  - class: org.yamcs.tctm.DataLinkInitialiser
  - class: org.yamcs.archive.XtceTmRecorder
  - class: org.yamcs.archive.ParameterRecorder
  - class: org.yamcs.archive.AlarmRecorder
  - class: org.yamcs.archive.EventRecorder
  - class: org.yamcs.archive.ReplayServer
  - class: org.yamcs.archive.IndexServer
  - class: org.yamcs.parameter.SystemParametersCollector
    args:
      provideJvmVariables: true
      provideFsVariables: true
  - class: org.yamcs.ProcessorCreatorService
    args: 
      name: "realtime"
      type: "realtime" 
  - class: org.yamcs.archive.CommandHistoryRecorder
  - class: org.yamcs.parameterarchive.ParameterArchive
    args:
      realtimeFiller: 
        enabled: false
      backFiller:
        enabled: true
        warmupTime: 60
  - class: org.yamcs.simulation.simulator.SimulatorCommander
    args:
      telnet:
        port: 10023
      tctm:
          tmPort: 10015
          tcPort: 10025
          losPort: 10115
          tm2Port: 10016
      # Simulator can send some packets to test the performance of Yamcs. 
      # Make sure the yamcs.simulator.yaml, mdb section contains a database generator for these packets, such that they are processed by Yamcs
      # if numPackets is greater than 0, the simulator will send <numPackets> packets of size <packetSize> at each <interval> (in ms)
      perfTest: 
        numPackets: 0 
        packetSize: 1476 #length of the performance testing packets
        interval: 10 
      frame: #send TM UDP frames 
          tmPort: 10017
          tmHost: "localhost"
          type: AOS # AOS, TM or USLP #TM
          tmFrameLength: 512 
          tmFrameFreq: 0 # #10 how many frames to send per second. If the simulator does not produce so many frames, it will send idle frames

#name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used
#tmIndexer: 

dataLinks:
  # The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
  # distributes them on the the three streams like the TCP links above
  # This works as alternative to the TCP links 
  - name: UDP_FRAME_IN
    #enabledAtStartup: false   #myAddition
    class: org.yamcs.tctm.UdpTmFrameLink
    args:
       port: 10017 #UDP port to listen to
       #one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
       frameType: "TM"
       spacecraftId: 0xAB
       frameLength: 512
       frameHeaderErrorControlPresent: false
       insertZoneLength: 0
       errorCorrection: NONE #CRC16 #NONE, CRC16 or CRC32 (only for USLP)
       virtualChannels:
          - vcId: 0
            ocfPresent: true
            service: "PACKET"  #supported services for AOS are M_PDU, B_PDU or VCA_SDU
            maxPacketLength: 2048
            packetPreprocessorClassName: org.yamcs.tctm.UdpTmDataLink
            packetPreprocessorArgs:
                stream: "tm2_realtime"
                port: 10017
            stream: "tm_realtime" #which yamcs stream to put the data on

          #vcId 63 is reserved for idle data and it does not have to be defined

mdb: scos-mib

#Configuration for streams created at server startup
streamConfig:
        tm:
         - name: "tm_realtime"
        cmdHist: ["cmdhist_realtime", "cmdhist_dump"]
        event: ["events_realtime", "events_dump"]
        param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"]
        alarm: ["alarms_realtime"]
        tc: ["tc_realtime"]
        sqlFile: "etc/extra_streams.sql"

So if we take out the org.yamcs.simulation.SimulatorCommander section, the same error pops up again. The same applies if we use the default packetPreprocessorClassname, with the right arguments. Even if instead of extra_streams.sql we use tc_streams.sql

Any idea on what could be the cause would be greatly appreciated.

xpromache commented 5 years ago

Ah, this is a bug caused by having some invalid times. We have to fix it but just to get rid of it, please delete all the files from /storage/yamcs-data/*

On Fri, Jun 7, 2019 at 6:27 PM claudio20497 notifications@github.com wrote:

After doing some modifications, we stumbled upon this error:

Jun 07 18:14:18.276 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284) org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: java.lang.NullPointerException org.yamcs.yarch.rocksdb.RdbPartitionManager.addPartitionByTimeAndValue(RdbPartitionManager.java:110) org.yamcs.yarch.rocksdb.RdbPartitionManager.readPartitions(RdbPartitionManager.java:47) org.yamcs.yarch.rocksdb.RdbStorageEngine.loadTable(RdbStorageEngine.java:84) org.yamcs.yarch.YarchDatabaseInstance.loadTables(YarchDatabaseInstance.java:148) org.yamcs.yarch.YarchDatabaseInstance.(YarchDatabaseInstance.java:96) org.yamcs.yarch.YarchDatabase.getInstance(YarchDatabase.java:75) org.yamcs.StreamInitializer.(StreamInitializer.java:46) org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:37) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762)

which throws no matter the config we use. Anyway our yamcs.sumulator.yaml is:

services:

  • class: org.yamcs.tctm.DataLinkInitialiser
  • class: org.yamcs.archive.XtceTmRecorder
  • class: org.yamcs.archive.ParameterRecorder
  • class: org.yamcs.archive.AlarmRecorder
  • class: org.yamcs.archive.EventRecorder
  • class: org.yamcs.archive.ReplayServer
  • class: org.yamcs.archive.IndexServer
  • class: org.yamcs.parameter.SystemParametersCollector args: provideJvmVariables: true provideFsVariables: true
  • class: org.yamcs.ProcessorCreatorService args: name: "realtime" type: "realtime"
  • class: org.yamcs.archive.CommandHistoryRecorder
  • class: org.yamcs.parameterarchive.ParameterArchive args: realtimeFiller: enabled: false backFiller: enabled: true warmupTime: 60
  • class: org.yamcs.simulation.simulator.SimulatorCommander args: telnet: port: 10023 tctm: tmPort: 10015 tcPort: 10025 losPort: 10115 tm2Port: 10016

    Simulator can send some packets to test the performance of Yamcs.

    Make sure the yamcs.simulator.yaml, mdb section contains a database generator for these packets, such that they are processed by Yamcs

    if numPackets is greater than 0, the simulator will send packets of size at each (in ms)

    perfTest: numPackets: 0 packetSize: 1476 #length of the performance testing packets interval: 10 frame: #send TM UDP frames tmPort: 10017 tmHost: "localhost" type: AOS # AOS, TM or USLP #TM tmFrameLength: 512 tmFrameFreq: 0 # #10 how many frames to send per second. If the simulator does not produce so many frames, it will send idle frames

    name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used#tmIndexer:

    dataLinks:

    The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and

    distributes them on the the three streams like the TCP links above

    This works as alternative to the TCP links

  • name: UDP_FRAME_IN

    enabledAtStartup: false #myAddition

    class: org.yamcs.tctm.UdpTmFrameLink args: port: 10017 #UDP port to listen to

    one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)

    frameType: "TM" spacecraftId: 0xAB frameLength: 512 frameHeaderErrorControlPresent: false insertZoneLength: 0 errorCorrection: NONE #CRC16 #NONE, CRC16 or CRC32 (only for USLP) virtualChannels:

    • vcId: 0 ocfPresent: true service: "PACKET" #supported services for AOS are M_PDU, B_PDU or VCA_SDU maxPacketLength: 2048 packetPreprocessorClassName: org.yamcs.tctm.UdpTmDataLink packetPreprocessorArgs: stream: "tm2_realtime" port: 10017 stream: "tm_realtime" #which yamcs stream to put the data on

      vcId 63 is reserved for idle data and it does not have to be defined

      mdb: scos-mib

      Configuration for streams created at server startupstreamConfig:

      tm:

      • name: "tm_realtime" cmdHist: ["cmdhist_realtime", "cmdhist_dump"] event: ["events_realtime", "events_dump"] param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"] alarm: ["alarms_realtime"] tc: ["tc_realtime"] sqlFile: "etc/extra_streams.sql"

So if we take out the org.yamcs.simulation.SimulatorCommander section, the same error pops up again. The same applies if we use the default packetPreprocessorClassname, with the right arguments. Even if instead of extra_streams.sql we use tc_streams.sql

Any idea on what could be the cause would be greatly appreciated.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR4PBZ3GGT77QM35GYDPZKEARA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGKNGQ#issuecomment-499951258, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6ARZRBD6ZYAKTLO3BECDPZKEARANCNFSM4HP2SDXA .

ClaudMor commented 5 years ago

Ok, we'll do it.

Before the last nullPointerException issue, we had this problem (quoting from previous comment):

So I noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not. If the packets were sent to and received by those VCs, then they would be put on the configurated streams and passed to the server (right?). if it is right, at the moment it may not happen because the configuration of those does not reflect our data (i think). Is there a way to know what the other possible config values are?

EDIT: if i go to monitor->events (in the web page), i see a lot of Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number) under the "message" column, from type and source UdpTmFrameLink.

We noticed that the class that throws this exception is TmTranferFrame, which has a method parseData in it which has not yet been implemented. This is the link to the class: https://github.com/yamcs/yamcs/blob/master/yamcs-core/src/main/java/org/yamcs/tctm/ccsds/TmTransferFrame.java We think this could be a blocking issue. Would changing the packetProcessorClassName inside the UDP_FRAME_IN dataLink solve the problem? Or how else may this be solved?

Thanks a lot for your support

xpromache commented 5 years ago

The method in the TmTransferFrame is a leftover method from an old implementation, it's not used and I removed it. The class that decodes the frames is TmFrameDecoder, I think this one throws the CRC error. Are you sure your frame has a CRC? I think this is optional: https://public.ccsds.org/Pubs/132x0b2.pdf (on page 4-2)

On Sat, Jun 8, 2019 at 3:14 PM claudio20497 notifications@github.com wrote:

Ok we'll do it.

when it worked, we had this isssue:

So i noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not. If the packets were sent to and received by those vcs, then they would be put on the configurated streams and passed to the server (right?). if it is right, at the moment it may not happen because the configuration of those is does not reflect our data (i think). Is there a way to know what the other possible config values are?

EDIT: if i go to monitor->events (in the web page), i see a lot of Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number) under the "message" column, from type and source UdpTmFrameLink.

We noticed that the class that throws this exception is TmTranferFrame, which has a method parseData in it which has not yet been implemented. This is the link to the class:

https://github.com/yamcs/yamcs/blob/master/yamcs-core/src/main/java/org/yamcs/tctm/ccsds/TmTransferFrame.java We think this could be a blocking issue. Would changing the packetProcessorClassName inside the UDP_FRAME_IN dataLink solve the problem? Or how else may this be solved?

Thanks a lot for your support

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR7U5BWH42EXWZFSOCDPZOWDNA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXHUNLA#issuecomment-500123308, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR66CYFGAFB5MACEXT3PZOWDNANCNFSM4HP2SDXA .

ClaudMor commented 5 years ago

Thanks, now it works fine (it starts, but we still have those warnings about crc). Yes, we have crc, but we'll remove it if not supported. Please let us know.

xpromache commented 5 years ago

The CRC should work, the error you get is that the CRC computation does not match the value in the frame therefore all frames are discarded. That can either be because the CRC it reads is not from the correct location (wrong frame size) or that the algorithm is bad. If you could send me the binary of a frame I could have a look, but if you cannot, you have to debug it yourself (just take the first frame from your file and compute the CRC using some other tool).

On Mon, Jun 10, 2019 at 9:37 AM claudio20497 notifications@github.com wrote:

Thanks, now it works fine (it starts, but we still have those warnings about crc). Yes, we have crc, but we'll remove it if not supported. Please let us know.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR7DFV7PVHKPTBQ4TDDPZYAFBA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJE7ZQ#issuecomment-500322278, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR5XMIOELRHTVM4BU6DPZYAFBANCNFSM4HP2SDXA .

ClaudMor commented 5 years ago

Hello, thanks for the info.

So we solved the crc problem by setting errorCorrection to NONE (is it a solution? It doesn't give warnings anymore).

Now the In_Count of the UDP_FRAME_IN increases.

1.We noticed that although there are no evident crashes, if i try to connect the packet-viewer to the server, i get "Unsupported operation subscribe".

2.Moreover, the "latest value" of parameters does not update (or how may i check if any parameter actually does without having to click on each of them in the we interface? they're too many of them)

Do you know which could be the causes of these issues?

Our configuration:

services:
  - class: org.yamcs.tctm.DataLinkInitialiser
  #- class: org.yamcs.archive.XtceTmRecorder
  - class: org.yamcs.archive.ParameterRecorder
  - class: org.yamcs.archive.AlarmRecorder
  - class: org.yamcs.archive.EventRecorder
  - class: org.yamcs.archive.ReplayServer
  - class: org.yamcs.archive.IndexServer
  - class: org.yamcs.parameter.SystemParametersCollector
    args:
      provideJvmVariables: true
      provideFsVariables: true
  - class: org.yamcs.ProcessorCreatorService
    args: 
      name: "realtime"
      type: "realtime"
  - class: org.yamcs.archive.CommandHistoryRecorder
  - class: org.yamcs.parameterarchive.ParameterArchive
    args:
      realtimeFiller: 
        enabled: false
      backFiller:
        enabled: true
        warmupTime: 60

#name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used
#tmIndexer: 

dataLinks:
  - name: tm_realtime
    enabledAtStartup: true
    class: org.yamcs.tctm.TcpTmDataLink
    args:
      stream: tm_realtime
      host: localhost
      port: 10015
  # The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
  # distributes them on the the three streams like the TCP links above
  # This works as alternative to the TCP links 
  - name: UDP_FRAME_IN
    enabledAtStartup: true
    class: org.yamcs.tctm.UdpTmFrameLink
    args:
       port: 10017 #UDP port to listen to
       #one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
       frameType: "TM"
       spacecraftId: (spacecraftID)
       frameLength: (frameLength)
       frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames
       insertZoneLength: 0
       errorCorrection: NONE #NONE, CRC16 or CRC32 (only for USLP)
       virtualChannels:
          - vcId: 1
            ocfPresent: true
            service: "PACKET"  #supported services for AOS are PACKET(M_PDU), B_PDU or VCA_SDU
            maxPacketLength: 2048
            stripEncapsulationHeader: true # drop the header before passing the packet to the preprocessor
            packetPreprocessorClassName: org.yamcs.tctm.GenericPacketPreprocessor
            packetPreprocessorArgs:
                timestampOffset: 2 #where to read the 8 bytes timestamp offset from
                seqCountOffset: 10 #where to read the 4 bytes sequence count from
                errorDetection: #last two bytes are used for the error detection
                   type: "CRC-16-CCIIT"  #"16-SUM"   #"CRC-16-CCIIT" 
            stream: "tm_realtime" #"tm2_realtime"

             #the other vcs are identical

mdb: scos-mib

#Configuration for streams created at server startup
streamConfig:
  tm:
    - name: "tm_realtime"
    - name: "tm2_realtime"
     # rootContainer: "/YSS/SIMULATOR/tm2_container"
    - name: "tm_dump"
    - name: "tc_sim"
    - name: "tc_tse"
  cmdHist: ["cmdhist_realtime", "cmdhist_dump"]
  event: ["events_realtime", "events_dump"]
  param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"]
  alarm: ["alarms_realtime"]
  tc: ["tc_realtime"]
  sqlFile: "etc/extra_streams.sql"
xpromache commented 5 years ago

Does the counter for VC 1 in the data links section increase? Check also for some relevant events (in the web interface monitoring -> events). I see you still have activated CRC at packet level, maybe that doesn't match.

The packet viewer is curious it cannot connect. Which version of the server and packet viewer do you use? How did you install them?

On Mon, Jun 10, 2019 at 3:07 PM claudio20497 notifications@github.com wrote:

Hello, thanks for the info.

So we solved the crc problem by setting errorCorrection to NONE (is it a solution? It doesn't give warnings anymore).

Now the In_Count of the UDP_FRAME_IN increases.

1.We noticed that although there are no evident crashes, if i try to connect the packet-viewer to the server, i get "Unsupported operation subscribe".

2.Moreover, the "latest value" of parameters does not update (or how may i check if any parameter actually does without having to click on each of them in the we interface? they're too many of them)

Do you know which could be the causes of these issues?

Our configuration:

services:

  • class: org.yamcs.tctm.DataLinkInitialiser

    - class: org.yamcs.archive.XtceTmRecorder

  • class: org.yamcs.archive.ParameterRecorder
  • class: org.yamcs.archive.AlarmRecorder
  • class: org.yamcs.archive.EventRecorder
  • class: org.yamcs.archive.ReplayServer
  • class: org.yamcs.archive.IndexServer
  • class: org.yamcs.parameter.SystemParametersCollector args: provideJvmVariables: true provideFsVariables: true
  • class: org.yamcs.ProcessorCreatorService args: name: "realtime" type: "realtime"
  • class: org.yamcs.archive.CommandHistoryRecorder
  • class: org.yamcs.parameterarchive.ParameterArchive args: realtimeFiller: enabled: false backFiller: enabled: true warmupTime: 60

    name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used#tmIndexer:

    dataLinks:

  • name: tm_realtime enabledAtStartup: true class: org.yamcs.tctm.TcpTmDataLink args: stream: tm_realtime host: localhost port: 10015

    The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and

    distributes them on the the three streams like the TCP links above

    This works as alternative to the TCP links

  • name: UDP_FRAME_IN enabledAtStartup: true class: org.yamcs.tctm.UdpTmFrameLink args: port: 10017 #UDP port to listen to

    one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)

    frameType: "TM" spacecraftId: (spacecraftID) frameLength: (frameLength) frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames insertZoneLength: 0 errorCorrection: NONE #NONE, CRC16 or CRC32 (only for USLP) virtualChannels:

    • vcId: 1 ocfPresent: true service: "PACKET" #supported services for AOS are PACKET(M_PDU), B_PDU or VCA_SDU maxPacketLength: 2048 stripEncapsulationHeader: true # drop the header before passing the packet to the preprocessor packetPreprocessorClassName: org.yamcs.tctm.GenericPacketPreprocessor packetPreprocessorArgs: timestampOffset: 2 #where to read the 8 bytes timestamp offset from seqCountOffset: 10 #where to read the 4 bytes sequence count from errorDetection: #last two bytes are used for the error detection type: "CRC-16-CCIIT" #"16-SUM" #"CRC-16-CCIIT" stream: "tm_realtime" #"tm2_realtime"

       #the other vcs are identical

      mdb: scos-mib

      Configuration for streams created at server startupstreamConfig:

      tm:

      • name: "tm_realtime"
      • name: "tm2_realtime"

        rootContainer: "/YSS/SIMULATOR/tm2_container"

      • name: "tm_dump"
      • name: "tc_sim"
      • name: "tc_tse" cmdHist: ["cmdhist_realtime", "cmdhist_dump"] event: ["events_realtime", "events_dump"] param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"] alarm: ["alarms_realtime"] tc: ["tc_realtime"] sqlFile: "etc/extra_streams.sql"

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR23A7J3XEZU4KND2NLPZZGZHA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJ2KEY#issuecomment-500409619, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6ARYEMK652PI7VF6VJPLPZZGZHANCNFSM4HP2SDXA .