Open ClaudMor opened 5 years ago
Hello, can you please describe what you want to achieve?
Hello,
Thanks for your time.
I have ccsds data and relative SCOS2K loaders. How may I inject them into yamcs, so that the ccsds data are interpreted and the result gets passed to the HTTP server, just as the simulator + yamcs-openmct-plugin work?
did you manage to install and run yamcs at all?
I suggest you first try to load the file in the packet-viewer bin/packet-viewer.sh - this should work without having the server started. You need first to make the mdb.yaml as described in the readme.
BTW: if you can share your MIB and test files, I can add some test cases on the public site. The MIB I'm using for testing cannot be made public.
Yes, we managed to start yamcs using "service yamcs-server start". We installed the scos2k plugin, the simulator and the yamcs-openmct plugin. The simulator and the yamcs-openmct plugin actually work as expected.
Unfortunately, we cannot share the MIB and test files.
Does the mdb.yaml file need to be in the etc folder of the server? Does any documentation regarding the use of the .sh scripts exist?
Thanks again
If you installed the server there is a file /opt/yamcs/etc/mdb.yaml on the server. You can edit that one to have the mdb defined.
Then you can start the /opt/yamcs/bin/packet-viewer.sh and do File->Open and in the XTCE DB on top, you should see the name of the config you added in mdb.yaml. Then you can select the CCSDS file and see if it loads. This will allow to verify that yamcs can decode properly the packets according to your MIB.
I must say that we (at SpaceApps) have never worked with the openmct and I don't know what it is doing. Do you want to load the file in the yamcs archive or somehow "play" it to simulate data coming in realtime? I don't know if openmct is able to access the Yamcs archive.
If i put the mdb.yaml file in /opt/yamcs/etc on the server and then start the server and the packet-viewer, the latter tells me that mdb.yaml is missing. But if i put the mdb.yaml in /opt/yamcs-client/etc, then it shows an error window stating "Cannot find a mapping for key ..."
Check the permission of the mdb.yaml, it should be readable by the user that executed packet-viewer.sh I just tried installing the yamcs 4.9.5 rpm, made a mdb.yaml and then I can start the packet viewer and load a file.
For the standalone packet-viewer, it is not necessary to run the server. After we get this running, we will be able to connect it to the server but first let's try in standalone mode.
If i put the mdb.yaml file in /opt/yamcs/etc on the server and then start the server and the packet-viewer, the latter tells me that mdb.yaml is missing. But if i put the mdb.yaml in /opt/yamcs-client/etc, then it shows an error window stating "Cannot find a mapping for key ..."
Sorry, now I realize that you start packet-viewer from /opt/yamcs-client, then yes you copy the mdb.yaml in the etc in that directory. Can you paste it here to see what's wrong?
I set permissions for all users but i keeps giving the same error. Anyway i was executing packet-viewer as root. So the command i'm giving is ./packet-viewer.sh -x scos-mib [telemetry-file-absolute-path]
. The [] are of course excluded.
I set permissions for all users but i keeps giving the same error. Anyway i was executing packet-viewer as root. So the command i'm giving is
./packet-viewer.sh -x scos-mib [telemetry-file-absolute-path]
. The [] are of course excluded.
can you show me your mdb.yaml?
the mdb.yaml is:
scos-mib:
typeOffset: 7
subTypeOffset: 8
epoch: "UNIX"
where path/to/ASCII is replaced with the path to the mib github replaces the "-" of yaml with a dot
I see, it's exactly the example from the readme file; can you then write the complete error?
No columns definition found in ui preferences
May 28 17:05:29.627 [1] org.yamcs.ui.packetviewer.PacketViewer loadLocalXtcedb SEVERE:cannot find a mapping for key 'TM': cannot find a mapping for key 'TM'
org.yamcs.YConfiguration.checkKey(YConfiguration.java:261)
org.yamcs.YConfiguration.getMap(YConfiguration.java:340)
org.yamcs.scos2k.TmMibLoader.
ah, sorry, the README file was not up2date. I just updated it now.
now it gives me:
No columns definition found in ui preferences (error line that reports an "unknown" parameter type. Unfortunately these lines also report confidential information so i can't paste them here) org.yamcs.scos2k.MibLoaderBits.getDataEncoding(MibLoaderBits.java:211) org.yamcs.scos2k.TmMibLoader.createParameterTypeNcateg(TmMibLoader.java:517) org.yamcs.scos2k.TmMibLoader.createParameterType(TmMibLoader.java:490) org.yamcs.scos2k.TmMibLoader.loadPfc(TmMibLoader.java:361) org.yamcs.scos2k.TmMibLoader.loadTelemetryParams(TmMibLoader.java:440) org.yamcs.scos2k.TmMibLoader.loadTelemetry(TmMibLoader.java:113) org.yamcs.scos2k.MibLoader.load(MibLoader.java:30) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:720) org.yamcs.xtceproc.XtceDbFactory$LoaderTree.load(XtceDbFactory.java:723) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:145) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:79) org.yamcs.xtceproc.XtceDbFactory.createInstanceByConfig(XtceDbFactory.java:67) org.yamcs.ui.packetviewer.PacketViewer.loadLocalXtcedb(PacketViewer.java:518) org.yamcs.ui.packetviewer.PacketViewer.openFile(PacketViewer.java:488) org.yamcs.ui.packetviewer.PacketViewer.main(PacketViewer.java:1173)
What could it be?
I see at that line the exception is "Unknonw parameter type (ptc,pfc)". But what are your parameter ptc,pfc (these should be just some numbers)?
Yes, we had a non-standard ptc,pfc . Probably it hasn't been implemented, we'll just remove it. Thank so much, we'll let you know how it goes!
Hello again,
We have another blocker issue: our MIB uses the symbol "<>" to mean the "different" symbol ("!="). Th problem is that the packet-viewer generates a java class from the file containing "<>" without translating "<>" into "!=", but it keeps the "<>" as it is in the generated java class. This causes a compilation error, because java does not recognize the symbol "<>" .
How could this be solved?
Thanks for your support
Can you please edit yamcs-scos2k/src/main/java/org/yamcs/scos2k/ol/Operation.java and change "<>" into "!=" on line 37: final static Operation DIFFER = new Operation("<>", COMP);
And then recompile (mvn install) and copy the jar into the yamcs lib.
Thanks, it worked.
we would like to signal to you that we noticed also that : OCF_CODIN=I and CDF_ELTYPE=F are not supported.
How could this be solved?
Moreover, after implementing your solution about the "<>", we were given an error about a null pointer exception, that we solved by changing
name -> parameters.get(name).ptype
in
name->parameters.get(mp.name).ptype
in file TmMibLoader.java (line 402) (was it a good move, or did we just cause a subtler chain of errors?)
thanks for your realtime support!!
OCF_CODIN=I is not supported Looking in the code I see it's supposed to work for numeric parameters but apparently yours is not numeric (most likely there is a bug somewhere else). Can you show the exact error?
CDF_ELTYPE=F I understand this means that the command argument is not editable. I'm not sure what the difference between this and the fixed area (CDF_ELTYPE=A) is. In any case, from what I can see, XTCE does not support read-only command arguments as such. What we could do is to create an abstract command and the real command can be inheriting from the abstract one and then it's possible to fix the arguments of the parent command.
Moreover, after implementing your solution about the "<>", we were given an error about a null pointer exception, that we solved by changing
name -> parameters.get(name).ptype
inname->parameters.get(mp.name).ptype
in file TmMibLoader.java (line 402) (was it a good move, or did we just cause a subtler chain of errors?)
I'm not sure your modification is good. The null pointer exception you get I think it's because the parameter that goes into the formula is not known. We should do better checking and error reporting in this case.
`
unfortunately we may not share the MIBs, but we noticed that one can solve the problem by again modifying the MIBs
Was not able to reproduce the error after correcting the other ones, probably it was a consequence of other mistakes.
Anyway now it seems to work (parameters appear in the packet-viewer window).
Next step would be to have something like the simulator: so i think we should pass the data to the server and create an instance. Could you please help us? How to do it? It seems i can't find exhaustive instructions in the documentation.
What you can do now is to modify the yamcs.simulator.yaml and add this under the data links:
dataLinks:
- name: file_import
class: org.yamcs.tctm.FilePollingTmDataLink
args:
stream: tm_realtime
incomingDir: "/path/to/incoming/data/dir"
deleteAfterImport: false
delayBetweenPackets: 1000
This will create you a data link called "file_import" with the data replayed from a file in the directory you specify (/path/to/incoming/dir). Make sure that directory has no other file inside than the one you have to replay. You can remove the other links or set enabledAtStartup: false to make sure they are not generating bogus data. Then you start or restart yamcs server (service yamcsd restart) and I suggest first connect with the web browser (http://yamcs-server-host:8090/ and go to the Links on the left to see that the data_in counter is increasing. Also check the logs for errors /opt/yamcs/log/yamcs-server.log.0
- This is the error stack for the OCF_CODIN=I (that we anyway "solved" by modifying the MIBs) `OCF_CODIN = I cannot be used for a parameter of type class org.yamcs.xtce.EnumeratedParameterType
I guess you have the OCF_INTER=U - that means the limit check has to be performed on the raw value. Currently this is ignored (all limits are performed on the engineering value) - so this is a bug in the loader.
I must say that the SCOS2k loader has been made as a proof of concept and used only on a limited MIB (which as we can see, was not using all features). If you decide to actually use it, I will go and fix those bugs. So please let me know...
Hello,
Yes, we will tell you more after the week-end! thank you for the moment, it seems to be working!
Hello,
When i said
Anyway now it seems to work (parameters appear in the packet-viewer window).
I was actually wrong: the packet viewer showed a few packets, but not all of them.
So we exported the data as hexdump and pointed the packet-viewer to the exported file.
Now the packet viewer seems to load all packets, but it gives the error:
No columns definition found in ui preferences Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Object.clone(Native Method) at org.yamcs.ui.packetviewer.ListPacket.<init>(ListPacket.java:31) at org.yamcs.ui.packetviewer.PacketsTable.packetReceived(PacketsTable.java:507) at org.yamcs.ui.packetviewer.PacketViewer$3.process(PacketViewer.java:701) at javax.swing.SwingWorker$3.run(SwingWorker.java:414) at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112) at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.run(SwingWorker.java:832) at sun.swing.AccumulativeRunnable.run(AccumulativeRunnable.java:112) at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.actionPerformed(SwingWorker.java:842) at javax.swing.Timer.fireActionPerformed(Timer.java:313) at javax.swing.Timer$DoPostEvent.run(Timer.java:245) at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:311) at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758) at java.awt.EventQueue.access$500(EventQueue.java:97) at java.awt.EventQueue$3.run(EventQueue.java:709) at java.awt.EventQueue$3.run(EventQueue.java:703) at java.security.AccessController.doPrivileged(Native Method) at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:74) at java.awt.EventQueue.dispatchEvent(EventQueue.java:728) at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:205) at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116) at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93) at java.awt.EventDispatchThread.run(EventDispatchThread.java:82) Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space
and then, when i click on a packet on the left tree, some warnings about corrupted files and SEQ_COUNT_JUMP appear.
Anyway we tried to start the server after modifying the dataLinks as you told us and the mdb.yaml in the server according to the readme. The file_import link is created and active, but the In_Count is zero. It keeps creating new server.out.log.0 files in /opt/yamcs/log, all of them just repeating the following error over and over:
Jun 03 18:12:17.136 [19] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /data/file Jun 03 18:12:17.136 [19] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /data/file: : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (exadecimalString) org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118) org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84) com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) com.google.common.util.concurrent.Callables$3.run(Callables.java:100) java.lang.Thread.run(Thread.java:748)
Where /data/file is the path to the data file. The exadecimalString is a exadecimal string longer than a packet header. It infact is a frame header. It seems that a packet header is expected. Is there an already implemeted way to extract packets from frames?
Thank you very much
Is your file containing packets or frames? What kind of frames?
Both packet viewer and FilePollingTmDataLink will only work with binary data, not hexadecimal.
Ok so the input needs to be a binary and NOT textual file. Our file contains frames with packets inside. If knowing the kind of frame is crucial to proceed, we will be able to tell you in 2-3 days at worst. If the the file must contain packets and not frames, is there an already-implemented way to extract packets from frames? (It gives me very similar errors in the server.logs if i point the server to the binary data and not the textual ones)
I don't know where you got the data from but I suspect it's either AOS frames: https://public.ccsds.org/Pubs/732x0b3e1.pdf or TM frames: https://public.ccsds.org/Pubs/132x0b2.pdf
The AOS frames start with first two bits 01 and the TM frame with 00. So you can probably tell from that.
In both cases, what you need to know is the frame size. Without it I'm afraid you cannot split the file in frames. But as you can see in the pdfs above, the first 10/12 bits is what they called Master Channel ID and that repeats in each frame. So you can try to guess the frame length from there.
Ok so thanks to your advice we discovered the kind of frame, and where to eventually split that. But we saw something strange: if we rely on the simulator, shouldn't the content of your readme go in /etc/yamcs.simulator.yaml, under the mdb section? (rather than in mdb.yaml?)
I ask this question because i noticed that if i delete all the mdb loaders in etc/yamcs.simulator.yaml and replace them with your readme content, while also keeping the mdb.yaml as before (so with your readme content copied inside), then intentionally-generated typos in the mdb.yaml are not reported as errors, while typos in the (for example) string "org.yamcs.scos2k.MibLoader" in yamcs.simulator.yaml are.
Moreover: if i do delete all the mdb loaders in etc/yamcs.simulator.yaml and replace them with your readme content, even without misspelling, it keeps telling me:
7:59:20.101 [1] [0;36mYamcsServer[0;0m yamcs 4.9.5, build 8424c81d32e33aa1ba804f913e2bf4bbe4cc588d 17:59:20.267 [1] [0;36mYamcsServer[0;0m Loading global service org.yamcs.web.HttpServer 17:59:20.454 [1] [0;36mYamcsServer[0;0m Loading global service org.yamcs.tse.TseCommander 17:59:20.473 [1] [0;36mYamcsServer[0;0m Loading online instance 'simulator' 17:59:20.488 [1] [0;36mYamcsServer[0;0m [0;31mSEVERE[0;0m Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284) org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: Cannot instantiate object from class org.yamcs.scos2k.MibLoader: java.lang.ClassNotFoundException: org.yamcs.scos2k.MibLoader org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:95) org.yamcs.xtceproc.XtceDbFactory.getLoaderTree(XtceDbFactory.java:510) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:104) org.yamcs.xtceproc.XtceDbFactory.getInstance(XtceDbFactory.java:620) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:98) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: java.lang.ClassNotFoundException: org.yamcs.scos2k.MibLoader java.net.URLClassLoader.findClass(URLClassLoader.java:382) java.lang.ClassLoader.loadClass(ClassLoader.java:424) sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) java.lang.ClassLoader.loadClass(ClassLoader.java:357) java.lang.Class.forName0(Native Method) java.lang.Class.forName(Class.java:264) org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:30) org.yamcs.xtceproc.XtceDbFactory.getLoaderTree(XtceDbFactory.java:510) org.yamcs.xtceproc.XtceDbFactory.createInstance(XtceDbFactory.java:104) org.yamcs.xtceproc.XtceDbFactory.getInstance(XtceDbFactory.java:620) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:98) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762)
while the yamcs.simulator.yaml is:
`services:
perfTest: numPackets: 0 packetSize: 1476 #length of the performance testing packets interval: 10 frame: #send TM UDP frames tmPort: 10017 tmHost: "localhost" type: AOS # AOS, TM or USLP tmFrameLength: #do we have to put here our frame length? tmFrameFreq: 10 # how many frames to send per second. If the simulator does not produce so many frames, it will send idle frames
dataLinks:
class: org.yamcs.tctm.FilePollingTmDataLink args: stream: tm_realtime incomingDir: "/path/to/data" #where /path/to/data is substituted with our pah to data deleteAfterImport: false delayBetweenPackets: 1000
mdb:
TC:
# default size in bytes of the size tag for variable length strings and bytestrings
vblParamLengthBytes: 0
#TM specific settings
TM:
vblParamLengthBytes: 1
# byte offset from beginning of the packet where the type and subType are read from
typeOffset: 7
subTypeOffset: 8
streamConfig: tm:
Thanks again, hope to able to give you more precise answers.
sorry for the confusion. Basically in yamcs.simulator.yaml you either put an entire config like you did, or you put
mdb: key-from-mdb-yaml
where key-from-mdb-yaml is the key of a config from mdb.yaml (e.g. scos-mib)
The only thing is that the packet viewer can work only with an mdb defined in the mdb.yaml, not in the yamcs.instance.yaml which is a config read by the yamcs server only.
The error you get is related to not finding the org.yamcs.scos2k.MibLoader which means that probably the yamcs-scos2k jar is not in the lib or lib/ext directory of the yamcs server (/opt/yamcs/lib/ normally). Or maybe the jar is not readable by the user that runs yamcs.
Thanks for solving the "org.yamcs.scos2k.MibLoader not found" exception! So if we just leave
mdb: scos-mib
in yamcs.simulator.yaml, it throws us:
Jun 06 10:14:51.355 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: Unknown sequence container: /YSS/SIMULATOR/tm2_container
org.yamcs.StreamConfig.<init>(StreamConfig.java:80)
org.yamcs.StreamConfig.lambda$getInstance$0(StreamConfig.java:34)
java.util.HashMap.computeIfAbsent(HashMap.java:1127)
org.yamcs.StreamConfig.getInstance(StreamConfig.java:34)
org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:51)
org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:39)
org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99)
org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
Jun 06 10:14:51.356 [11] org.yamcs.yarch.rocksdb.RDBFactory shutdown FINE:shutting down, closing 1 databases under /storage/yamcs-data/simulator.rdb: []
while if i leave the other MIBs,
mdb:
# Configuration of the active loaders
# Valid loaders are: sheet, xtce or fully qualified name of the class
- type: "sheet"
spec: "mdb/simulator-ccsds.xls"
subLoaders:
- type: "sheet"
spec: "mdb/landing.xls"
- type: "sheet"
spec: "mdb/GS-tmtc.xls"
- type: "org.yamcs.xtce.TseLoader"
subLoaders:
- type: "sheet"
spec: "mdb/tse/simulator.xls"
#Loads the performance testing mission database
- type: "org.yamcs.simulator.PerfMdbLoader"
args:
numPackets: 18
packetSize: 1476
- type: "org.yamcs.scos2k.MibLoader"
args:
path: "/path/to/ASCII"
epoch: "UNIX"
#TC specific settings
TC:
# default size in bytes of the size tag for variable length strings and bytestrings
vblParamLengthBytes: 0
#TM specific settings
TM:
vblParamLengthBytes: 1
# byte offset from beginning of the packet where the type and subType are read from
typeOffset: 7
subTypeOffset: 8
it throws:
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /path/to/data
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /path/to/data : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (unreadable string)
org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118)
org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84)
com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
com.google.common.util.concurrent.Callables$3.run(Callables.java:100)
java.lang.Thread.run(Thread.java:748)
over and over. So the solution is to keep the second configuration and split the data in frames, right?
The reason for this error is that in the streamConfig section you have
tm:
...
- name: "tm2_realtime"
rootContainer: "/YSS/SIMULATOR/tm2_container"
which means that data that comes on the tm2_realtime stream has to be processed according to the tm2_container. This container doesn't exist since you use your own MDB.
So, just remove that "rootContainer" - meaning that the tm2_realtime data will be processed according to the default root container from your MDB. In reality you could remove the tm2_realtime all together but you have to also remove it in the data link section.
FYI: in Yamcs a Datalink puts data into a Stream from where it is processed by a Processor or recorded in the so called stream database (based on rocksdb).
Thanks, it worked.
Now , using
mdb: scos-mib
it gives the second error:
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run INFO:Injecting the content of /path/to/data
Jun 06 10:23:17.374 [26] org.yamcs.tctm.FilePollingTmDataLink[simulator] run WARNING:Got IOException while reading from /path/to/data : java.io.IOException: ASCII header too long, probably not a PaCTS archive file: (unreadable string)
org.yamcs.tctm.TmFileReader.readPacket(TmFileReader.java:118)
org.yamcs.tctm.FilePollingTmDataLink.run(FilePollingTmDataLink.java:84)
com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
com.google.common.util.concurrent.Callables$3.run(Callables.java:100)
java.lang.Thread.run(Thread.java:748)
ok, that's because the file you put in there is ASCII and it thinks it's another format (Pacts - some old EGSE software, we have in fact to remove it from the open-source yamcs).
I'm afraid you have now to sort out the data format problem, and I see two options:
you extract packets out of frames into a different file and put that file to be read by yamcs.
you use the yamcs framing support but unfortunately we cannot read frames from files at this moment. What you can do is to make a small python script to send the frames one by one via UDP. In the simulator configuration, yamcs expects frames via UDP on port 10017 Make sure you tune the parameters
args:
port: 10017 #UDP port to listen to
#one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
frameType: "AOS"
spacecraftId: 0xAB
frameLength: 512
frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames
insertZoneLength: 0
errorCorrection: CRC16 #NONE, CRC16 or CRC32 (only for USLP)
according to your data.
Ok. Which format should the frames be sent in? An array of bytes?
One UDP packet = one frame. An array of bytes if you want but binary please no hexadecimal or any formatting.
On Thu, Jun 6, 2019 at 3:14 PM claudio20497 notifications@github.com wrote:
Ok. Which format should the frames be sent in? An array of bytes?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR35TP4AZKTBV36G7UTPZEEUHA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXCZM3Q#issuecomment-499488366, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR5F7AVBPY3VOQWYJZTPZEEUHANCNFSM4HP2SDXA .
Ok. So we no longer use the dataLink FilePollingTmDataLink, right?
That's right. That one only works with packets.
On Thu, Jun 6, 2019 at 3:43 PM claudio20497 notifications@github.com wrote:
Ok. So we no longer use the dataLink FilePollingTmDataLink, right?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR4GBOR5W5JB6U5QTU3PZEH65A5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXC4AYY#issuecomment-499499107, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR26HFKVN4LFZWFG4UTPZEH65ANCNFSM4HP2SDXA .
Hello,
So now data arrives to the simulator, but it fails to send them to the server. We have frame containing ccsds data. we use, as packetpreprocessor,
packetPreprocessorClassName: org.yamcs.tctm.CcsdsPacketInputStream
with no arguments, as the documentation (https://www.yamcs.org/yamcs/javadoc/org/yamcs/tctm/CcsdsPacketInputStream.html) seems to suggest. Is it correct? Looking at the error below, it appears to need an argument.
Jun 06 19:07:33.012 [1] org.yamcs.tctm.ccsds.VirtualChannelPacketHandler[simulator] <init> SEVERE:Cannot instantiate the packet preprocessor: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:75)
org.yamcs.tctm.ccsds.VirtualChannelPacketHandler.<init>(VirtualChannelPacketHandler.java:56)
org.yamcs.tctm.ccsds.TmManagedParameters.createVcHandlers(TmManagedParameters.java:65)
org.yamcs.tctm.ccsds.MasterChannelFrameHandler.<init>(MasterChannelFrameHandler.java:52)
org.yamcs.tctm.UdpTmFrameLink.<init>(UdpTmFrameLink.java:57)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
org.yamcs.tctm.DataLinkInitialiser.createDataLink(DataLinkInitialiser.java:98)
org.yamcs.tctm.DataLinkInitialiser.<init>(DataLinkInitialiser.java:57)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
org.yamcs.YamcsServer.createService(YamcsServer.java:705)
org.yamcs.YamcsServer.createServices(YamcsServer.java:145)
org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:101)
org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
Jun 06 19:07:33.012 [1] org.yamcs.YamcsServer createServices SEVERE:Cannot create service org.yamcs.tctm.DataLinkInitialiser, with arguments null: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
Jun 06 19:07:33.013 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: Cannot find a constructor for class 'org.yamcs.tctm.CcsdsPacketInputStream' and arguments (java.lang.String)
org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:75)
org.yamcs.tctm.ccsds.VirtualChannelPacketHandler.<init>(VirtualChannelPacketHandler.java:56)
org.yamcs.tctm.ccsds.TmManagedParameters.createVcHandlers(TmManagedParameters.java:65)
org.yamcs.tctm.ccsds.MasterChannelFrameHandler.<init>(MasterChannelFrameHandler.java:52)
org.yamcs.tctm.UdpTmFrameLink.<init>(UdpTmFrameLink.java:57)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.yamcs.utils.YObjectLoader.loadObject(YObjectLoader.java:78)
org.yamcs.tctm.DataLinkInitialiser.createDataLink(DataLinkInitialiser.java:98)
org.yamcs.tctm.DataLinkInitialiser.<init>(DataLinkInitialiser.java:57)
Moreover, do you think that this is the right preprocessor to read ccsds frame and process ccsds packet to be sent to the yamcs server?
Thanks a lot
that CccsdsPacketInputStream is splitting a stream into packets. That doesn't work with frames (it works if you have a file of packets or if you send packets via a tcp connection).
What I suggested is that you use the config from the simulator that receives frames via UDP. Then you don't need to split any stream because UDP is providing the splitting.
If you look here: https://github.com/yamcs/yamcs/blob/master/yamcs-simulation/etc/yamcs.simulator.yaml you see that the tm2_realtime and tm_realtime use the PacketInputStream (although it is not explicitley configured for tm_realtime).
The UDP_FRAME_IN link uses UDP (so no stream splitter) and provides the functionality for extracting the packets from the frames. I think this is the one you want but you need to send somehow the frames to yamcs. Note that the simulator has nothing to do in your setup and I suggest you remove it from the config file (remove the section org.yamcs.simulation.simulator.SimulatorCommander) because it generates packets and frames which you don't want.
Ok thanks.
So i noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not.
If the packets were sent to and received by those vcs, then they would be put on the configurated streams and passed to the server (right?).
if it is right,
at the moment it may not happen because the configuration of those is does not reflect our data (i think). Is there a way to know what the other possible config values are?
EDIT: if i go to monitor->events (in the web page), i see a lot of
Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number)
under the "message" column, from type and source UdpTmFrameLink.
After doing some modifications, we stumbled upon this error:
Jun 07 18:14:18.276 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED
org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284)
org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
Caused by: java.lang.NullPointerException
org.yamcs.yarch.rocksdb.RdbPartitionManager.addPartitionByTimeAndValue(RdbPartitionManager.java:110)
org.yamcs.yarch.rocksdb.RdbPartitionManager.readPartitions(RdbPartitionManager.java:47)
org.yamcs.yarch.rocksdb.RdbStorageEngine.loadTable(RdbStorageEngine.java:84)
org.yamcs.yarch.YarchDatabaseInstance.loadTables(YarchDatabaseInstance.java:148)
org.yamcs.yarch.YarchDatabaseInstance.<init>(YarchDatabaseInstance.java:96)
org.yamcs.yarch.YarchDatabase.getInstance(YarchDatabase.java:75)
org.yamcs.StreamInitializer.<init>(StreamInitializer.java:46)
org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:37)
org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99)
org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63)
org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82)
org.yamcs.YamcsServer.createInstance(YamcsServer.java:523)
org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279)
org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362)
org.yamcs.YamcsServer.main(YamcsServer.java:762)
which throws no matter the config we use. Anyway our yamcs.sumulator.yaml is:
services:
- class: org.yamcs.tctm.DataLinkInitialiser
- class: org.yamcs.archive.XtceTmRecorder
- class: org.yamcs.archive.ParameterRecorder
- class: org.yamcs.archive.AlarmRecorder
- class: org.yamcs.archive.EventRecorder
- class: org.yamcs.archive.ReplayServer
- class: org.yamcs.archive.IndexServer
- class: org.yamcs.parameter.SystemParametersCollector
args:
provideJvmVariables: true
provideFsVariables: true
- class: org.yamcs.ProcessorCreatorService
args:
name: "realtime"
type: "realtime"
- class: org.yamcs.archive.CommandHistoryRecorder
- class: org.yamcs.parameterarchive.ParameterArchive
args:
realtimeFiller:
enabled: false
backFiller:
enabled: true
warmupTime: 60
- class: org.yamcs.simulation.simulator.SimulatorCommander
args:
telnet:
port: 10023
tctm:
tmPort: 10015
tcPort: 10025
losPort: 10115
tm2Port: 10016
# Simulator can send some packets to test the performance of Yamcs.
# Make sure the yamcs.simulator.yaml, mdb section contains a database generator for these packets, such that they are processed by Yamcs
# if numPackets is greater than 0, the simulator will send <numPackets> packets of size <packetSize> at each <interval> (in ms)
perfTest:
numPackets: 0
packetSize: 1476 #length of the performance testing packets
interval: 10
frame: #send TM UDP frames
tmPort: 10017
tmHost: "localhost"
type: AOS # AOS, TM or USLP #TM
tmFrameLength: 512
tmFrameFreq: 0 # #10 how many frames to send per second. If the simulator does not produce so many frames, it will send idle frames
#name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used
#tmIndexer:
dataLinks:
# The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
# distributes them on the the three streams like the TCP links above
# This works as alternative to the TCP links
- name: UDP_FRAME_IN
#enabledAtStartup: false #myAddition
class: org.yamcs.tctm.UdpTmFrameLink
args:
port: 10017 #UDP port to listen to
#one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
frameType: "TM"
spacecraftId: 0xAB
frameLength: 512
frameHeaderErrorControlPresent: false
insertZoneLength: 0
errorCorrection: NONE #CRC16 #NONE, CRC16 or CRC32 (only for USLP)
virtualChannels:
- vcId: 0
ocfPresent: true
service: "PACKET" #supported services for AOS are M_PDU, B_PDU or VCA_SDU
maxPacketLength: 2048
packetPreprocessorClassName: org.yamcs.tctm.UdpTmDataLink
packetPreprocessorArgs:
stream: "tm2_realtime"
port: 10017
stream: "tm_realtime" #which yamcs stream to put the data on
#vcId 63 is reserved for idle data and it does not have to be defined
mdb: scos-mib
#Configuration for streams created at server startup
streamConfig:
tm:
- name: "tm_realtime"
cmdHist: ["cmdhist_realtime", "cmdhist_dump"]
event: ["events_realtime", "events_dump"]
param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"]
alarm: ["alarms_realtime"]
tc: ["tc_realtime"]
sqlFile: "etc/extra_streams.sql"
So if we take out the org.yamcs.simulation.SimulatorCommander section, the same error pops up again. The same applies if we use the default packetPreprocessorClassname, with the right arguments. Even if instead of extra_streams.sql we use tc_streams.sql
Any idea on what could be the cause would be greatly appreciated.
Ah, this is a bug caused by having some invalid times. We have to fix it but just to get rid of it, please delete all the files from /storage/yamcs-data/*
On Fri, Jun 7, 2019 at 6:27 PM claudio20497 notifications@github.com wrote:
After doing some modifications, we stumbled upon this error:
Jun 07 18:14:18.276 [1] org.yamcs.YamcsServer main SEVERE:Could not start Yamcs Server: java.lang.IllegalStateException: Expected the service to be INITIALIZED, but the service has FAILED org.yamcs.YamcsInstanceService.checkCurrentState(YamcsInstanceService.java:284) org.yamcs.YamcsInstanceService.awaitInitialized(YamcsInstanceService.java:142) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:83) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) Caused by: java.lang.NullPointerException org.yamcs.yarch.rocksdb.RdbPartitionManager.addPartitionByTimeAndValue(RdbPartitionManager.java:110) org.yamcs.yarch.rocksdb.RdbPartitionManager.readPartitions(RdbPartitionManager.java:47) org.yamcs.yarch.rocksdb.RdbStorageEngine.loadTable(RdbStorageEngine.java:84) org.yamcs.yarch.YarchDatabaseInstance.loadTables(YarchDatabaseInstance.java:148) org.yamcs.yarch.YarchDatabaseInstance.
(YarchDatabaseInstance.java:96) org.yamcs.yarch.YarchDatabase.getInstance(YarchDatabase.java:75) org.yamcs.StreamInitializer. (StreamInitializer.java:46) org.yamcs.StreamInitializer.createStreams(StreamInitializer.java:37) org.yamcs.YamcsServerInstance.doInit(YamcsServerInstance.java:99) org.yamcs.YamcsInstanceService.initAsync(YamcsInstanceService.java:63) org.yamcs.YamcsServerInstance.init(YamcsServerInstance.java:82) org.yamcs.YamcsServer.createInstance(YamcsServer.java:523) org.yamcs.YamcsServer.createGlobalServicesAndInstances(YamcsServer.java:279) org.yamcs.YamcsServer.setupYamcsServer(YamcsServer.java:362) org.yamcs.YamcsServer.main(YamcsServer.java:762) which throws no matter the config we use. Anyway our yamcs.sumulator.yaml is:
services:
- class: org.yamcs.tctm.DataLinkInitialiser
- class: org.yamcs.archive.XtceTmRecorder
- class: org.yamcs.archive.ParameterRecorder
- class: org.yamcs.archive.AlarmRecorder
- class: org.yamcs.archive.EventRecorder
- class: org.yamcs.archive.ReplayServer
- class: org.yamcs.archive.IndexServer
- class: org.yamcs.parameter.SystemParametersCollector args: provideJvmVariables: true provideFsVariables: true
- class: org.yamcs.ProcessorCreatorService args: name: "realtime" type: "realtime"
- class: org.yamcs.archive.CommandHistoryRecorder
- class: org.yamcs.parameterarchive.ParameterArchive args: realtimeFiller: enabled: false backFiller: enabled: true warmupTime: 60
- class: org.yamcs.simulation.simulator.SimulatorCommander args: telnet: port: 10023 tctm: tmPort: 10015 tcPort: 10025 losPort: 10115 tm2Port: 10016
Simulator can send some packets to test the performance of Yamcs.
Make sure the yamcs.simulator.yaml, mdb section contains a database generator for these packets, such that they are processed by Yamcs
if numPackets is greater than 0, the simulator will send
packets of size at each (in ms) perfTest: numPackets: 0 packetSize: 1476 #length of the performance testing packets interval: 10 frame: #send TM UDP frames tmPort: 10017 tmHost: "localhost" type: AOS # AOS, TM or USLP #TM tmFrameLength: 512 tmFrameFreq: 0 # #10 how many frames to send per second. If the simulator does not produce so many frames, it will send idle frames
name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used#tmIndexer:
dataLinks:
The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
distributes them on the the three streams like the TCP links above
This works as alternative to the TCP links
name: UDP_FRAME_IN
enabledAtStartup: false #myAddition
class: org.yamcs.tctm.UdpTmFrameLink args: port: 10017 #UDP port to listen to
one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
frameType: "TM" spacecraftId: 0xAB frameLength: 512 frameHeaderErrorControlPresent: false insertZoneLength: 0 errorCorrection: NONE #CRC16 #NONE, CRC16 or CRC32 (only for USLP) virtualChannels:
vcId: 0 ocfPresent: true service: "PACKET" #supported services for AOS are M_PDU, B_PDU or VCA_SDU maxPacketLength: 2048 packetPreprocessorClassName: org.yamcs.tctm.UdpTmDataLink packetPreprocessorArgs: stream: "tm2_realtime" port: 10017 stream: "tm_realtime" #which yamcs stream to put the data on
vcId 63 is reserved for idle data and it does not have to be defined
mdb: scos-mib
Configuration for streams created at server startupstreamConfig:
tm:
- name: "tm_realtime" cmdHist: ["cmdhist_realtime", "cmdhist_dump"] event: ["events_realtime", "events_dump"] param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"] alarm: ["alarms_realtime"] tc: ["tc_realtime"] sqlFile: "etc/extra_streams.sql"
So if we take out the org.yamcs.simulation.SimulatorCommander section, the same error pops up again. The same applies if we use the default packetPreprocessorClassname, with the right arguments. Even if instead of extra_streams.sql we use tc_streams.sql
Any idea on what could be the cause would be greatly appreciated.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR4PBZ3GGT77QM35GYDPZKEARA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXGKNGQ#issuecomment-499951258, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6ARZRBD6ZYAKTLO3BECDPZKEARANCNFSM4HP2SDXA .
Ok, we'll do it.
Before the last nullPointerException issue, we had this problem (quoting from previous comment):
So I noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not. If the packets were sent to and received by those VCs, then they would be put on the configurated streams and passed to the server (right?). if it is right, at the moment it may not happen because the configuration of those does not reflect our data (i think). Is there a way to know what the other possible config values are?
EDIT: if i go to monitor->events (in the web page), i see a lot of Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number) under the "message" column, from type and source UdpTmFrameLink.
We noticed that the class that throws this exception is TmTranferFrame, which has a method parseData in it which has not yet been implemented. This is the link to the class: https://github.com/yamcs/yamcs/blob/master/yamcs-core/src/main/java/org/yamcs/tctm/ccsds/TmTransferFrame.java We think this could be a blocking issue. Would changing the packetProcessorClassName inside the UDP_FRAME_IN dataLink solve the problem? Or how else may this be solved?
Thanks a lot for your support
The method in the TmTransferFrame is a leftover method from an old implementation, it's not used and I removed it. The class that decodes the frames is TmFrameDecoder, I think this one throws the CRC error. Are you sure your frame has a CRC? I think this is optional: https://public.ccsds.org/Pubs/132x0b2.pdf (on page 4-2)
On Sat, Jun 8, 2019 at 3:14 PM claudio20497 notifications@github.com wrote:
Ok we'll do it.
when it worked, we had this isssue:
So i noticed that although the In_Count of UDP_FRAME_IN actually increases, the In_Counts of the virtual channels do not. If the packets were sent to and received by those vcs, then they would be put on the configurated streams and passed to the server (right?). if it is right, at the moment it may not happen because the configuration of those is does not reflect our data (i think). Is there a way to know what the other possible config values are?
EDIT: if i go to monitor->events (in the web page), i see a lot of Error processing frame: org.yamcs.tctm.ccsds.CorruptedFrameException: Bad CRC computed: (number) in the frame: (different number) under the "message" column, from type and source UdpTmFrameLink.
We noticed that the class that throws this exception is TmTranferFrame, which has a method parseData in it which has not yet been implemented. This is the link to the class:
https://github.com/yamcs/yamcs/blob/master/yamcs-core/src/main/java/org/yamcs/tctm/ccsds/TmTransferFrame.java We think this could be a blocking issue. Would changing the packetProcessorClassName inside the UDP_FRAME_IN dataLink solve the problem? Or how else may this be solved?
Thanks a lot for your support
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR7U5BWH42EXWZFSOCDPZOWDNA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXHUNLA#issuecomment-500123308, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR66CYFGAFB5MACEXT3PZOWDNANCNFSM4HP2SDXA .
Thanks, now it works fine (it starts, but we still have those warnings about crc). Yes, we have crc, but we'll remove it if not supported. Please let us know.
The CRC should work, the error you get is that the CRC computation does not match the value in the frame therefore all frames are discarded. That can either be because the CRC it reads is not from the correct location (wrong frame size) or that the algorithm is bad. If you could send me the binary of a frame I could have a look, but if you cannot, you have to debug it yourself (just take the first frame from your file and compute the CRC using some other tool).
On Mon, Jun 10, 2019 at 9:37 AM claudio20497 notifications@github.com wrote:
Thanks, now it works fine (it starts, but we still have those warnings about crc). Yes, we have crc, but we'll remove it if not supported. Please let us know.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR7DFV7PVHKPTBQ4TDDPZYAFBA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJE7ZQ#issuecomment-500322278, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6AR5XMIOELRHTVM4BU6DPZYAFBANCNFSM4HP2SDXA .
Hello, thanks for the info.
So we solved the crc problem by setting errorCorrection to NONE (is it a solution? It doesn't give warnings anymore).
Now the In_Count of the UDP_FRAME_IN increases.
1.We noticed that although there are no evident crashes, if i try to connect the packet-viewer to the server, i get "Unsupported operation subscribe".
2.Moreover, the "latest value" of parameters does not update (or how may i check if any parameter actually does without having to click on each of them in the we interface? they're too many of them)
Do you know which could be the causes of these issues?
Our configuration:
services:
- class: org.yamcs.tctm.DataLinkInitialiser
#- class: org.yamcs.archive.XtceTmRecorder
- class: org.yamcs.archive.ParameterRecorder
- class: org.yamcs.archive.AlarmRecorder
- class: org.yamcs.archive.EventRecorder
- class: org.yamcs.archive.ReplayServer
- class: org.yamcs.archive.IndexServer
- class: org.yamcs.parameter.SystemParametersCollector
args:
provideJvmVariables: true
provideFsVariables: true
- class: org.yamcs.ProcessorCreatorService
args:
name: "realtime"
type: "realtime"
- class: org.yamcs.archive.CommandHistoryRecorder
- class: org.yamcs.parameterarchive.ParameterArchive
args:
realtimeFiller:
enabled: false
backFiller:
enabled: true
warmupTime: 60
#name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used
#tmIndexer:
dataLinks:
- name: tm_realtime
enabledAtStartup: true
class: org.yamcs.tctm.TcpTmDataLink
args:
stream: tm_realtime
host: localhost
port: 10015
# The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
# distributes them on the the three streams like the TCP links above
# This works as alternative to the TCP links
- name: UDP_FRAME_IN
enabledAtStartup: true
class: org.yamcs.tctm.UdpTmFrameLink
args:
port: 10017 #UDP port to listen to
#one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
frameType: "TM"
spacecraftId: (spacecraftID)
frameLength: (frameLength)
frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames
insertZoneLength: 0
errorCorrection: NONE #NONE, CRC16 or CRC32 (only for USLP)
virtualChannels:
- vcId: 1
ocfPresent: true
service: "PACKET" #supported services for AOS are PACKET(M_PDU), B_PDU or VCA_SDU
maxPacketLength: 2048
stripEncapsulationHeader: true # drop the header before passing the packet to the preprocessor
packetPreprocessorClassName: org.yamcs.tctm.GenericPacketPreprocessor
packetPreprocessorArgs:
timestampOffset: 2 #where to read the 8 bytes timestamp offset from
seqCountOffset: 10 #where to read the 4 bytes sequence count from
errorDetection: #last two bytes are used for the error detection
type: "CRC-16-CCIIT" #"16-SUM" #"CRC-16-CCIIT"
stream: "tm_realtime" #"tm2_realtime"
#the other vcs are identical
mdb: scos-mib
#Configuration for streams created at server startup
streamConfig:
tm:
- name: "tm_realtime"
- name: "tm2_realtime"
# rootContainer: "/YSS/SIMULATOR/tm2_container"
- name: "tm_dump"
- name: "tc_sim"
- name: "tc_tse"
cmdHist: ["cmdhist_realtime", "cmdhist_dump"]
event: ["events_realtime", "events_dump"]
param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"]
alarm: ["alarms_realtime"]
tc: ["tc_realtime"]
sqlFile: "etc/extra_streams.sql"
Does the counter for VC 1 in the data links section increase? Check also for some relevant events (in the web interface monitoring -> events). I see you still have activated CRC at packet level, maybe that doesn't match.
The packet viewer is curious it cannot connect. Which version of the server and packet viewer do you use? How did you install them?
On Mon, Jun 10, 2019 at 3:07 PM claudio20497 notifications@github.com wrote:
Hello, thanks for the info.
So we solved the crc problem by setting errorCorrection to NONE (is it a solution? It doesn't give warnings anymore).
Now the In_Count of the UDP_FRAME_IN increases.
1.We noticed that although there are no evident crashes, if i try to connect the packet-viewer to the server, i get "Unsupported operation subscribe".
2.Moreover, the "latest value" of parameters does not update (or how may i check if any parameter actually does without having to click on each of them in the we interface? they're too many of them)
Do you know which could be the causes of these issues?
Our configuration:
services:
- class: org.yamcs.tctm.DataLinkInitialiser
- class: org.yamcs.archive.XtceTmRecorder
- class: org.yamcs.archive.ParameterRecorder
- class: org.yamcs.archive.AlarmRecorder
- class: org.yamcs.archive.EventRecorder
- class: org.yamcs.archive.ReplayServer
- class: org.yamcs.archive.IndexServer
- class: org.yamcs.parameter.SystemParametersCollector args: provideJvmVariables: true provideFsVariables: true
- class: org.yamcs.ProcessorCreatorService args: name: "realtime" type: "realtime"
- class: org.yamcs.archive.CommandHistoryRecorder
- class: org.yamcs.parameterarchive.ParameterArchive args: realtimeFiller: enabled: false backFiller: enabled: true warmupTime: 60
name of the class that is used for the TM index. If not specified, org.yamcs.archive.GenericCccsdsTmIndex is used#tmIndexer:
dataLinks:
- name: tm_realtime enabledAtStartup: true class: org.yamcs.tctm.TcpTmDataLink args: stream: tm_realtime host: localhost port: 10015
The link below receives AOS frames (CCSDS 732.0-B-3) on three virtual channels and
distributes them on the the three streams like the TCP links above
This works as alternative to the TCP links
name: UDP_FRAME_IN enabledAtStartup: true class: org.yamcs.tctm.UdpTmFrameLink args: port: 10017 #UDP port to listen to
one of AOS (CCSDS 732.0-B-3), TM (CCSDS 132.0-B-2) or USLP (CCSDS 732.1-B-1)
frameType: "TM" spacecraftId: (spacecraftID) frameLength: (frameLength) frameHeaderErrorControlPresent: true #2 bytes of reed-solomon (10,6) in the header of all frames insertZoneLength: 0 errorCorrection: NONE #NONE, CRC16 or CRC32 (only for USLP) virtualChannels:
vcId: 1 ocfPresent: true service: "PACKET" #supported services for AOS are PACKET(M_PDU), B_PDU or VCA_SDU maxPacketLength: 2048 stripEncapsulationHeader: true # drop the header before passing the packet to the preprocessor packetPreprocessorClassName: org.yamcs.tctm.GenericPacketPreprocessor packetPreprocessorArgs: timestampOffset: 2 #where to read the 8 bytes timestamp offset from seqCountOffset: 10 #where to read the 4 bytes sequence count from errorDetection: #last two bytes are used for the error detection type: "CRC-16-CCIIT" #"16-SUM" #"CRC-16-CCIIT" stream: "tm_realtime" #"tm2_realtime"
#the other vcs are identical
mdb: scos-mib
Configuration for streams created at server startupstreamConfig:
tm:
- name: "tm_realtime"
- name: "tm2_realtime"
rootContainer: "/YSS/SIMULATOR/tm2_container"
- name: "tm_dump"
- name: "tc_sim"
- name: "tc_tse" cmdHist: ["cmdhist_realtime", "cmdhist_dump"] event: ["events_realtime", "events_dump"] param: ["pp_realtime", "pp_tse", "sys_param", "proc_param"] alarm: ["alarms_realtime"] tc: ["tc_realtime"] sqlFile: "etc/extra_streams.sql"
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/yamcs/yamcs-scos2k/issues/1?email_source=notifications&email_token=AAG6AR23A7J3XEZU4KND2NLPZZGZHA5CNFSM4HP2SDXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXJ2KEY#issuecomment-500409619, or mute the thread https://github.com/notifications/unsubscribe-auth/AAG6ARYEMK652PI7VF6VJPLPZZGZHANCNFSM4HP2SDXA .
Hello,
I'm absolutely new to Yamcs. From the notes it is clear how to load the SCOS2K tables. But how does one load the actual ccsds data?
Thanks in advance