Closed kkretzschmar closed 8 years ago
jconejero
commented
8 years ago Hi Klaus,
This is fantastic! I am excited with this new module, and can't wait to release it. Yes, please post instructions to test it on OS X.
Thank you so much for your great work. This is a very important contribution to PixInsight.
kkretzschmar
commented
8 years ago Hopefully the qualitiy is good enough ;) ...
Yes, please post instructions to test it on OS X.
A short overview of the INDI system can be found here
http://indilib.org/about/discover-indi.html
You can run the INDI controlling system in two configurations:
The INDI server was originally written for Linux OS, but in the meantime there exists an implementation for OS X and on Windows there is an INDI wrapper for the ASCOM framework.
I think the INIDI server is mostly used on Linux OS, currently INDI running on ARM computers like raspberry pi is coming more and more.
When you have successfully installed the INDI server on the machine where you plug-in your devices, then start the indiserver with
indiserver -v -l
then you will see two entries in the INDI Devices section of the client interface GUI. The simulator devices are for testing purposes only. If you want to use a DSLR use the indi_gphoto_ccd device driver.
The option
-
-m
Use the INDI Controller Client to connect to the server and the devices. Connect the client to one or several devices by selecting one (several) device(s) and press the Connect button. Repeat this step if you want to connect to several devices. After successful connection the server sends all device properties to the client.
With the Controller Client you are able to change the parameters of the devices in a general manner. However when nyou want to trigger frame acquisitions use the INDICCDFrame process. This module is a convenience app to perform frame acquisitions. It expects a runningINDI Controller Client * with a connected CCD device. Choose the CCD device from the list of connected devices, specify the exposure duration and the number of frames and start the exposure and other parameters. The server will send the images to the client and the *INDICCDFrame process will display it in PI.
Let me know if you have any trouble/problems with setting up the whole story ... unfortunately it is quite complex.
Klaus
2016-01-15 9:29 GMT+01:00 Juan Conejero notifications@github.com:
Hi Klaus,
This is fantastic! I am excited with this new module, and can't wait to release it. Yes, please post instructions to test it on OS X.
Thank you so much for your great work. This is a very important contribution to PixInsight.
— Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-171900474.
jconejero
commented
8 years ago Hi Klaus,
I have now completed a first revision and testing work on your INDIClient module. There are extensive source code changes and improvements, especially in INDICCDFrame and INDIDeviceController. I have made also a moderate revision work on INDIMount, but this process still needs a lot of work.
INDICCDFrame is now ready for release IMHO. It works remarkably well. As you'll see, it is now a fully instantiable process with complete global execution capabilities, perfectly integrated with the platform. INDIDeviceController is also instantiable and works very well, although the interface still needs some work. Overall, I think we are very close to a first release, perhaps excluding INDIMount until we manage to develop it further.
The documentation is now on the Reference-Documentation repository.
There are specific questions and topics we need to review together; I'll post them in a separate comment.
I hope you'll agree with the changes, please let me know otherwise.
kkretzschmar
commented
8 years ago Hi Juan,
As you'll see, it is now a fully instantiable process with complete global execution capabilities, perfectly integrated with the platform. INDIDeviceController is also instantiable and works >very well, although the interface still needs some work. Wow, thanks! Yes this is something I disregarded a bit since I was not sure how the two modules should behave together. I guess, it is now possible to use the INDICCDFrame process from Javacript as well?
Overall, I think we are very close to a first release, perhaps excluding INDIMount Yes, not releasing the INDIMount module was my plan anyway. The GUI has to be adapted to the standard, and it would be cool to reuse some function from your stargenerator process. The module is only a prototype of what a INDIMount module could look like. I use it for mount control in my backyard observatory, I see many opportunities for automation together with the image processing capabilities of the PixInsight platform. I would appreciate any discussion about how we can proceed here.
There are specific questions and topics we need to review together; I'll post them in a separate comment. Yes of course.
I hope you'll agree with the changes, Absolutely. Thanks again. I'll now better understand, how the modules can be better integrated into the PI platform.
2016-03-18 14:55 GMT+01:00 Juan Conejero notifications@github.com:
Hi Klaus,
I have now completed a first revision and testing work on your INDIClient module. There are extensive source code changes and improvements, especially in INDICCDFrame and INDIDeviceController. I have made also a moderate revision work on INDIMount, but this process still needs a lot of work.
INDICCDFrame is now ready for release IMHO. It works remarkably well. As you'll see, it is now a fully instantiable process with complete global execution capabilities, perfectly integrated with the platform. INDIDeviceController is also instantiable and works very well, although the interface still needs some work. Overall, I think we are very close to a first release, perhaps excluding INDIMount until we manage to develop it further.
The documentation is now on the Reference-Documentation repository.
There are specific questions and topics we need to review together; I'll post them in a separate comment.
I hope you'll agree with the changes, please let me know otherwise.
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-198372499
kkretzschmar
commented
8 years ago Hi Juan, I tested your modifications and it works, thanks. I noticed that some features are missing:
1.) Setting the name of the image file on server side. Server side upload is important if you want to save the upload time to the remote client which can be quite long for big images ...
2.) Furthermore when aborting the process an abort exposure command should be sent to the INDI server.
And finally I remember why I decided not to fully implement the PI-process interface for the INDICCDFrame and the INDIMount process. In my opinion it is not optimal that the process GUI of the INDIFrame process and the INDIclient process is blocked when acquiring an exposure. The INDI server is a real service and can handle commands in parallel even during taking exposures.
For example, lets assume we started a long exposure and after some time you noticed that the name of the image file on the server is wrong. Then you want to be able to change that name during the running exposure.
I know that my previous version of the INDICCDFrame did not offer the possibiity to control it via the javascript runtime. But this was decided by purpose. For controlling the INDI server via the javascript engine it is sufficient to use the INDIDeviceControlller. My idea was to have the INDIDeviceController as the only one fully instantiable process and the other processes act more like non-modal GUI processes (like a monitor process) with some additional functionalities:
Those apps are not really useful for javascript automation. For automation in javascript the INDIDeviceController is sufficient. Actually my first version of the INDICCDFrame and the INDIMount process was completely implemented in javascript using the INDIDeviceController client only. I didn't like it because Javascript is slow and there are only modal Dialogs possible ... Maybe this has changed with QT5...
Maybe I am missing a fundamental architectural point ...
What do you think?
2016-03-18 16:40 GMT+01:00 Klaus Kretzschmar kkretzschmar@googlemail.com:
Hi Juan,
As you'll see, it is now a fully instantiable process with complete global execution capabilities, perfectly integrated with the platform. INDIDeviceController is also instantiable and works >very well, although the interface still needs some work. Wow, thanks! Yes this is something I disregarded a bit since I was not sure how the two modules should behave together. I guess, it is now possible to use the INDICCDFrame process from Javacript as well?
Overall, I think we are very close to a first release, perhaps excluding INDIMount Yes, not releasing the INDIMount module was my plan anyway. The GUI has to be adapted to the standard, and it would be cool to reuse some function from your stargenerator process. The module is only a prototype of what a INDIMount module could look like. I use it for mount control in my backyard observatory, I see many opportunities for automation together with the image processing capabilities of the PixInsight platform. I would appreciate any discussion about how we can proceed here.
There are specific questions and topics we need to review together; I'll post them in a separate comment. Yes of course.
I hope you'll agree with the changes, Absolutely. Thanks again. I'll now better understand, how the modules can be better integrated into the PI platform.
2016-03-18 14:55 GMT+01:00 Juan Conejero notifications@github.com:
Hi Klaus,
I have now completed a first revision and testing work on your INDIClient module. There are extensive source code changes and improvements, especially in INDICCDFrame and INDIDeviceController. I have made also a moderate revision work on INDIMount, but this process still needs a lot of work.
INDICCDFrame is now ready for release IMHO. It works remarkably well. As you'll see, it is now a fully instantiable process with complete global execution capabilities, perfectly integrated with the platform. INDIDeviceController is also instantiable and works very well, although the interface still needs some work. Overall, I think we are very close to a first release, perhaps excluding INDIMount until we manage to develop it further.
The documentation is now on the Reference-Documentation repository.
There are specific questions and topics we need to review together; I'll post them in a separate comment.
I hope you'll agree with the changes, please let me know otherwise.
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-198372499
jconejero
commented
8 years ago Hi Klaus,
Thank you. I am trying to integrate your module with the PixInsight platform correctly and efficiently. The version I have uploaded is just a first, and still rough, approximation. I think it starts to point in the right direction, but there's still a lot of work to do, and many pending design options and decisions.
1.) Setting the name of the image file on server side.
Of course. Note that INDICCDFrameInstance has a p_serverFileNameTemplate parameter for this purpose, whose functionality is not yet implemented. My idea is to write a simple interpreter to build template strings similar to printf(). For example:
%F will be replaced with the filter name %e replaced with the exposure time in seconds %T replaced with the UTC time of acquisition %f replaced with the frame type %b replaced with binnings %n replaced with the current frame number
and so on. So for example:
"%f_B%bE%e%n"
would generate a template (UPLOAD_PREFIX) such as:
"LIGHT_B1x1_E30.00_012"
for the 12th frame in the series. This is very easy to implement and very useful and flexible IMO. What do you think?
2.) Furthermore when aborting the process an abort exposure command should be sent to the INDI server.
Yep. This has to be included in INDICCDFrameInstance::ExecuteGlobal(). This can be done very easily with a try/catch block, I'll implement it.
For example, lets assume we started a long exposure and after some time you noticed that the name of the image file on the server is wrong. Then you want to be able to change that name during the running exposure.
Yes, I have been aware that this functionality was being lost in a formal process implementation.
There are several possibilities to implement this:
Personally I would follow the first option, since it provides the maximum flexibility.
The second option is always possible, of course, and obviously gives access to the whole INDI server functionality (for example, one could change filters in the middle of an acquisition sequence, or change the sensor temperature, etc).
For controlling the INDI server via the javascript engine it is sufficient to use the INDIDeviceControlller. My idea was to have the INDIDeviceController as the only one fully instantiable process and the other processes act more like non-modal GUI processes (like a monitor process)
In PixInsight we always go for the maximum flexibility and efficiency, and this is especially true in this case, since we are working on a critical component that opens a whole new window of functionality to the platform.
There is no reason to make "toy" tools with INDICCDFrame and INDIMount. IMO both processes should be first-class citizens in the standard tool set.
The possibility to instantiate INDICCDFrame in PJSR to run an acquisition task is simply wonderful. It is simple, easy-to-use, efficient, and works perfectly (once we implement the lacking bits). The same will be true for INDIMount to move a telescope in a simple and direct way from JavaScript, when we implement it as an instantiable process. Note that with these two processes one can write an automated image acquisition task that can be integrated in a complete image acquisition and prepocessing pipeline.
Even more importantly, the beauty of these processes is that they are completely independent on the INDI server and the INDI protocol. Both processes could use a completely different underlying implementation without any changes visible from outside, because they behave as abstract, self-contained components. This is a key concept in the design and architecture of PixInsight.
Of course, INDIDeviceController is also instantiable to provide full control over the INDI server. This is great, and gives us even more power and flexibility. The good news is that this is perfectly compatible with INDICCDFrame and INDIMount being also instantiable abstract processes.
So we can have the best of both worlds: abstract funcional components that integrate elegantly with the platform (INDICCDFrame, INDIMount, ...), and a low-level component (INDIDeviceController) that provides the maximum control over a specific implementation. What else? :)
kkretzschmar
commented
8 years ago Hi Juan,
The version I have uploaded is just a first, and still rough, approximation. Ah, OK.
My idea is to write a simple interpreter to build template strings similar to printf() ... What do you think? Nice idea! So your idea is to overwrite the _XX suffix template on server side? That makes perfectly sense because this server-side template seems to be driver dependent. Some drivers use a Image_XX and some a Image_XXX template leading to different ranges for different devices which is not so good ...
Yes, I have been aware that this functionality was being lost in a formal process implementation. There are several possibilities to implement this: ... Personally I would follow the first option, since it provides the maximum flexibility. Yes, I agree. And with the INDIDeviceControllerInstance at hand you have control over the full INDI server functionality too.
This is a key concept in the design and architecture of PixInsight. I fully agree.
So we can have the best of both worlds: abstract funcional components that integrate elegantly with the platform (INDICCDFrame, INDIMount, ...), and a low-level component (INDIDeviceController) that provides the maximum control over a specific implementation. What else? :) OK, got it. This is of course the perfect solution, and yes I agree, the Javascript developer should also benefit from the CCDFrame and Mount process.
What about the next steps. Do you already have an estimate over the remaining effort for the first release? Where can I support you? Shall I do some javascript testing, or should I start to work on the INDIMount process to make it fully instantiable?
Am 20.03.2016 um 09:54 schrieb Juan Conejero:
Hi Klaus,
Thank you. I am trying to integrate your module with the PixInsight platform correctly and efficiently. The version I have uploaded is just a first, and still rough, approximation. I think it starts to point in the right direction, but there's still a lot of work to do, and many pending design options and decisions.
1.) Setting the name of the image file on server side.
Of course. Note that INDICCDFrameInstance has a p_serverFileNameTemplate parameter for this purpose, whose functionality is not yet implemented. My idea is to write a simple interpreter to build template strings similar to printf(). For example:
%F will be replaced with the filter name %e replaced with the exposure time in seconds %T replaced with the UTC time of acquisition %f replaced with the frame type %b replaced with binnings %n replaced with the current frame number
and so on. So for example:
"%f_B%bE%e%n"
would generate a template (UPLOAD_PREFIX) such as:
"LIGHT_B1x1_E30.00_012"
for the 12th frame in the series. This is very easy to implement and very useful and flexible IMO. What do you think?
2.) Furthermore when aborting the process an abort exposure command should be sent to the INDI server.
Yep. This has to be included in INDICCDFrameInstance::ExecuteGlobal(). This can be done very easily with a try/catch block, I'll implement it.
For example, lets assume we started a long exposure and after some time you noticed that the name of the image file on the server is wrong. Then you want to be able to change that name during the running exposure.
Yes, I have been aware that this functionality was being lost in a formal process implementation.
There are several possibilities to implement this:
- Reimplement the "Start" and "Stop" buttons that you had included in in INDICCDFrameInterface, reusing the code you wrote based on timers. This would allow a user to run an exposure series in "interactive mode", in a way very similar to how a process like Blink works, for example. This has to be done very carefully, but is perfectly doable. Note that this is fully compatible with the formal global execution capabilities of INDICCDFrameInstance, which are much more important IMO.
- Run a second instance of the PixInsight Core application (or have it running already when you launch INDICCDFrame instance), then use INDIDeviceController.
- Implement a custom AbortProcess handler in INDICCDFrameInstance::ExecuteGlobal().
Personally I would follow the first option, since it provides the maximum flexibility.
The second option is always possible, of course, and obviously gives access to the whole INDI server functionality (for example, one could change filters in the middle of an acquisition sequence, or change the sensor temperature, etc).
For controlling the INDI server via the javascript engine it is sufficient to use the INDIDeviceControlller. My idea was to have the INDIDeviceController as the only one fully instantiable process and the other processes act more like non-modal GUI processes (like a monitor process)
In PixInsight we always go for the maximum flexibility and efficiency, and this is especially true in this case, since we are working on a critical component that opens a whole new window of functionality to the platform.
There is no reason to make "toy" tools with INDICCDFrame and INDIMount. IMO both processes should be first-class citizens in the standard tool set.
The possibility to instantiate INDICCDFrame in PJSR to run an acquisition task is simply wonderful. It is simple, easy-to-use, efficient, and works perfectly (once we implement the lacking bits). The same will be true for INDIMount to move a telescope in a simple and direct way from JavaScript, when we implement it as an instantiable process. Note that with these two processes one can write an automated image acquisition task that can be integrated in a complete image acquisition and prepocessing pipeline.
Even more importantly, the beauty of these processes is that they are completely independent on the INDI server and the INDI protocol. Both processes could use a completely different underlying implementation without any changes visible from outside, because they behave as abstract, self-contained components. This is a key concept in the design and architecture of PixInsight.
Of course, INDIDeviceController is also instantiable to provide full control over the INDI server. This is great, and gives us even more power and flexibility. The good news is that this is perfectly compatible with INDICCDFrame and INDIMount being also instantiable abstract processes.
So we can have the best of both worlds: abstract funcional components that integrate elegantly with the platform (INDICCDFrame, INDIMount, ...), and a low-level component (INDIDeviceController) that provides the maximum control over a specific implementation. What else? :)
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-198876229
jconejero
commented
8 years ago Hi Klaus,
What about the next steps. Do you already have an estimate over the remaining effort for the first release?
The module works well. I see no problem to release a first version next week, after Easter holidays. Maybe Thursday 31 or Friday May 1st.
should I start to work on the INDIMount process to make it fully instantiable?
I think we should release a simplified version of INDIMount without graphics, that is, without maps. Just a simple tool like INDICCDFrame, able to control a mount, and of course instantiable. Once we have this tool, the module is ready for release IMO.
Shall I do some javascript testing
Sure, It would be nice if we'd have a few demo scripts, which could be released along with the module.
I'll work on file name template support for INDICCDFrame during these days. I'm going to be "on vacation" for a few days, but fully online all the time. I'll return next Tuesday.
Be sure this module is going to cause a moderately strong earthquake. I know of a few companies that are going to be quite concerned... :)
kkretzschmar
commented
8 years ago Hi Juan,
The module works well. I see no problem to release a first version next week, after Easter holidays. Maybe Thursday 31 or Friday May 1st. OK, great!
I think we should release a simplified version of INDIMount without graphics, that is, without maps. Just a simple tool like INDICCDFrame, able to control a mount, and of course >instantiable. Once we have this tool, the module is ready for release IMO. OK, I can put the graphics part under a #ifdef so that I can use it for my own purpose until we have something better. I'll start work on making it instantiable.
Sure, It would be nice if we'd have a few demo scripts, which could be released along with the module. OK. Maybe a simple script for acquisition of dark bias and flat frames. I am not so deep in the theory of master frames, is there something (e.g. checks) which could make such a script even more valuable for the user. For example checking/calibrating that flat frames are not saturated etc ... I think the basic parameters will be of course exposure time, temperature, type of frame,... Are there some easy fornulas to determine the optimal exposure time, e.g. for bias or flat frames?
I'll work on file name template support for INDICCDFrame during these days. I'm going to be "on vacation" for a few days, but fully online all the time. I'll return next Tuesday. OK, great. Have a nice vacation, I'll also take some days off, I think there is no hurry ...
Be sure this module is going to cause a moderately strong earthquake. I know of a few companies that are going to be quite concerned... :) :) wasn't aware of that
Best regards, Klaus
2016-03-23 12:57 GMT+01:00 Juan Conejero notifications@github.com:
Hi Klaus,
What about the next steps. Do you already have an estimate over the remaining effort for the first release?
The module works well. I see no problem to release a first version next week, after Easter holidays. Maybe Thursday 31 or Friday May 1st.
should I start to work on the INDIMount process to make it fully instantiable?
I think we should release a simplified version of INDIMount without graphics, that is, without maps. Just a simple tool like INDICCDFrame, able to control a mount, and of course instantiable. Once we have this tool, the module is ready for release IMO.
Shall I do some javascript testing
Sure, It would be nice if we'd have a few demo scripts, which could be released along with the module.
I'll work on file name template support for INDICCDFrame during these days. I'm going to be "on vacation" for a few days, but fully online all the time. I'll return next Tuesday.
Be sure this module is going to cause a moderately strong earthquake. I know of a few companies that are going to be quite concerned... :)
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-200320501
cmilovic
commented
8 years ago Are there some easy fornulas to determine the optimal exposure time, e.g. for bias or flat frames?
Hi Klaus
First of all, congratulations for this module. :) It is a great addition to the PI platform. Thank you!
Now about the optimal exposure... For bias frames, you have to use the shortest exposure time avalaible for the device. Also you should ask the user to cover the scope. :) For flat frames, a simple rule of thumb is to have the mean intensities in the middle of the avalaible dinamic range. This means, for 16bits, roughly
Hope this helps.
Regards,
Carlos
PhD (C) - Engineering Sciences Department of Electrical Engineering
Software Developer PixInsight Development Team
Research Assistant Biomedical Imaging Center - PUC http://www.mri.cl
kkretzschmar
commented
8 years ago Hi Carlos,
First of all, congratulations for this module. :) It is a great addition to the PI platform. Thank you! Thanks, hope it will be useful for you. I see a big potential for further development especially when combining the powerful PI image analysing capabilities with controlling capabilities.
For bias frames, you have to use the shortest exposure time avalaible for the device. The INDI device driver should provide the info about the shortest possible exposure time, but I am not sure whether I already read it out from the server messages. I'll check, if not it will come in a later version ..
Also you should ask theuser to cover the scope. :) Yes, good point ;)
For flat frames, a simple rule of thumb is to have the mean intensities in the middle of the avalaible dinamic range. This means, for 16bits, roughly 32300. I think this easily to realize. It is possible to read out pixel properties, like pixel sizes or bits-per-pixel, from the INDI server.
Hope this helps. Yes absolutely, thanks!
Best regards, Klaus
2016-03-23 16:37 GMT+01:00 cmilovic notifications@github.com:
Are there some easy fornulas to determine the optimal exposure time, e.g. for bias or flat frames?
Hi Klaus
First of all, congratulations for this module. :) It is a great addition to the PI platform. Thank you!
Now about the optimal exposure... For bias frames, you have to use the shortest exposure time avalaible for the device. Also you should ask the user to cover the scope. :) For flat frames, a simple rule of thumb is to have the mean intensities in the middle of the avalaible dinamic range. This means, for 16bits, roughly
- But, in practice it could be a bit more difficult, since the ADC may work at 12bits or 14bits, and it may use the least relevant bits of the 16bit range. For example, a Canon camera tipically uses 14bits, whereas the maximum value reaches 16383 ADU. For this camera, you should expose to get a flat close to ~8200 ADU.
Hope this helps.
Regards,
Carlos
PhD (C) - Engineering Sciences Department of Electrical Engineering
Pontificia Universidad Católica de Chile
Software Developer PixInsight Development Team
http://www.pixinsight.com
Research Assistant Biomedical Imaging Center - PUC http://www.mri.cl
— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/PixInsight/PCL/pull/16#issuecomment-200399011
Hi Juan, I have now a stable version for a pre-release of the INDIClient and the CCDFrame module. I tested it on Linux only, but I will also test on windows. MacOS is currently difficult for me to test, since I have no Mac available and my Linux Laptop is too weak for VMs. If you also want to give it a try let me know and I'll send you a description how to setup the INDI server etc ...
Klaus