data source independent marker code

[felix-salfelder]

the marker code (particularly the move operation) is higly entangled with the double arrays contained in the class holding the data corresponding to a symbol in a .dat file (DatFile or something). this makes it impossible to encapsulate the dat parser into its own class.

ideally, the dat file contents should just be labelled forward iterable waves (or containers with waves), one idea is to copy over the relevant data to the graph instance and do the moving there. why I think this is optimal: because there are only a few pixels on the screen, and it also makes sense to select a subset of the data for plotting purposes. this attempt could solve both these issues (provided someone implements a selector).

maybe there's a suitable Qt class that implements line segments with user data. it might make sense to use a vector of those replacing Graph::ScrPoints. well, maybe.

[nvdl]

If we have to sort this out, it should be from the origin.

As a first step, we need a ".dat" to HDF5 converter.

Later, for compatibility with other simulators, we will need a "x-simulator to HDF5" converter.

I personally believe that text based simulation output is good for debugging purposes but not for efficient storage. It also affects the simulation speed with small timestamps unless one is using SSD.

Later, we will see how to configure "qucsator" to write out HDF5 directly.

I am voting for HDF5 but other possibilities can also be looked into.

[felix-salfelder]

If we have to sort this out, it should be from the origin.

at the origin, there is an interface, not a file format. file formats lock you into storing data in files. an interface to a data source (be it a dat file or a hdf file) can be used more flexibly.

Later, we will see how to configure "qucsator" to write out HDF5 directly.

hdf5 possibly is the right output format for qucsator. lets assume it really is. now other simulators will still produce other files... what if hdf6 comes out? will you rewrite qucs another time?

[nvdl]

at the origin, there is an interface, not a file format. file formats lock you into storing data in files. an interface to a data source (be it a dat file or a hdf file) can be used more flexibly.

Sorry, I cannot understand it properly.

what if hdf6 comes out?

People are still using HDF4. If HDF6 comes out, we may not need to new features or the upgrade should be minimal.

I am thinking of taking the whole file format thing out from "qucs-gui".

What if we use shared memory between "qucs-gui" and "qucsator" to share simulation data; no file conversion and disk I/O overhead. May be an issue for large simulation data and less memory. For large simulations, we can emulate memory region while still reading from a memory image file on disk?

Another possibility is to come up with a binary (and not text based) simulation output format.

Let's shout out all the possibilities before we converge.

BTW, how do you write that grey text block in the comments? :smile:

[felix-salfelder]

Sorry, I cannot understand it properly.

think of the interface as the base class of the parsers for various date formats.

class SimOutputBase { .. };

qucs-gui gets simulation data from an instance of this class. which could be either

class DatFileParser : public SimOutputBase { .. };

or

class HDFParser : public SimOutputBase { .. };

or anything else (added later, without changes on the qucs side).

I am thinking of taking the whole file format thing out from "qucs-gui".

so do I. but i think the result should be independent of a file format choice. (sorry, HDF5 is a file format, isn't it?).

BTW, how do you write that grey text block in the comments?

this is a greater character followed by a space at the beginning of the line.

[guitorri]

+1 for an abstract interface for data sources. +1 for supporting other file formats.

Right now the only file format that Qucs understants is its own (call it) qucsdata format. ra3xdh already wrote some other custom converters on the spice4qucs branch to convert ngspice/xyce output into qucsdata.

IMHO we should move all the read/write/conversion code to qucsconv (including new file formats). Turn it into a shared library and link it with qucs and qucsator. Doing so we will have all the flexibility we need. We just need to add option to qucsator and it will spit out whatever supported file format. Then qucs figure out the loading and drop the data into a nice and clean data container...

As I said elsewhere, I am not sure we should let qucs depend on qucsator (vice-versa). Modularity-wise and compilation-wise...

Having the data formats in a single place will ease the testing and creation of a clean interface. But... this is not really the topic of this issue...

[felix-salfelder]

IMHO we should move all the read/write/conversion code to qucsconv.

this is an interesting idea. currently i am doing a step-by-step transition from entangled diagram code to independent diagram code. this is more tricky than i had expected (for reasons such as this ticket). so really i prefer to finish the transition and then move the headers and maybe parsers to a shared location (be it qucsconv, i don't mind).

using the data from a simulator infrastructure might need some more thought until it can be used as an online converter (between qucsator and the output file), if at all. look at why gnucap does not (yet) support output plugins. i think it's best to write to a HDF5 structure within qucsator, and just dump into a file -- for now. it should be easy to pass a SimOutputBase view to that structure to a future qucsconv library or to some other frontend.

[nvdl]

so do I. but i think the result should be independent of a file format choice. (sorry, HDF5 is a file format, isn't it?).

As said, I was thinking about shared memory between "qucsator" and "qucs-gui". The shared memory will contain the data structures related to nodes, probes, etc. We hard code those structures once and possibly forever.

For other simulators, we need a converter and memory loader. But this approach has limitation of the system memory. I was trying to skip disk I/O; though not a big issue by now.

I feel it is OK if we go with HDF5 and ".dat" as the main supported formats for "qucs-gui". Other simulators' outputs can be converted to these formats and passed. Some work related to ".dat" is already done.

[felix-salfelder]

As said, I was thinking about shared memory between "qucsator" and "qucs-gui".

earlier, you wrote

for compatibility with other simulators, we will need a "x-simulator to HDF5" converter.

to me this sounded a bit like choosing a file format. can't we agree that none of qucs-gui should depend on the file format used (except for the parsers, of course).

We hard code those structures once and possibly forever.

Yes. except that we should only hardcode the interface.

For other simulators, we need a converter and memory loader.

we do not need a converter. we need to implement the interface in front of what the simulator produces. be that a file, a pointer to something else or a chunk of shared memory.

note that there is no 'other' in other simulators. the simulators should be treated alike.

I feel it is OK if we go with HDF5 and ".dat" as the main supported formats for "qucs-gui".

what do you mean by main? default maybe? i am fine with this. in the end, we want to interface simulators, not files.

lets get back to topic. i think we need .dat independent markers, no matter what comes next.

[nvdl]

Do we have IRC or other chat source where we can discuss it online?

lets get back to topic. i think we need .dat independent markers, no matter what comes next.

Forget about my previous entangled reply; was busy with work.

i think it's best to write to a HDF5 structure within qucsator, and just dump into a file -- for now

I agree with that. I will look into "qucsator" and add a switch that will make it dump the new format. I will give HDF5 the first shot; no prior experience though. I will communicate how the data is organized in HDF5 file from "qucsator" so that your class knows how to parse the file. This will only be done if there are many ways to store data in HDF5. Meanwhile, you get started or proceed with what has been already done and let me know how the class will load and represent the graphs and sweeps.

I personally feel, HDF5 will allow a query based data loading like SQL so we do not have to load all the stuff if not needed.

Meanwhile, I need help with understanding dependent and independent variables. Are these the only types in ".dat"? These variables are either real or complex. Right? Double precision; 64 bit? Any document?

Seems fine?

[felix-salfelder]

Do we have IRC or other chat source where we can discuss it online?

how about #qucs on oftc?

I personally feel, HDF5 will allow a query based data loading like SQL so we do not have to load all the stuff if not needed.

the data interface should/will support lazy loading, so that's good news, lets make use of it!

Right? Double precision; 64 bit?

in dat files, all values are implicitly complex. i think this is bad, but it's a start. it should be possible to add other types later on. thinking of different precision, reals, digital stuff.

there's some more, but nothing as important as having a working substitute for the dat file.

[in3otd]

guitorri registered #qucs on freenode.net some time ago but it is mostly unused, at present. But I agree it will make sense to use IRC to avoid sidetracking open issues.

[nvdl]

I am on "oftc" now; #qucs.

qucs on freenode.net seems dead.

[felix-salfelder]

what do you guys think of using Qt to take care of the plot style? nowadays, qt can draw paths through control nodes with a custom dashed style. the current code computes dash segment end points when loading data into a graph.

changing to the Qt painter would dramatically reduce code complexity. i'm not so sure about speed though. maybe Qt uses hardware acceleration of some sort, the current code certainly does not. after the change it will be easier to concentrate on the essential speed issues, like sample preselection on huge data sets.

(yes, i already implemented this out of curiosity, but a PR needs some more work)

[guitorri]

@felix-salfelder you are right. Each graphical object (or composite objected) should take care of its own painting on a paint method. The custom Viewpainter should be removed. Currently the paint/update of all object are queued and rendered sequentially, hence the slowness. Moving forward to QGraphics it uses a clever algorithm (binary space partitioning) working behind the scenes doing the update very efficiently. This is very fast. Goog for "40000 chips" Qt demo. It can also use an OpenGL mode. Other alternative is QML, which is supposed to be even faster, but requires a major rewriting in a different language.

If you can bring all the necessary code inside the paint method, it should be easy to port it forward when the graphical objects become a QGraphicsItem (owned by a QGraphicsScene, rendered in a QGraphicsView which is the main widget of the Schematic document).

Later today I will put up my latest WIP on the QGraphics so you can see what I mean with the above.

[felix-salfelder]

ok, see PR #297