Could not read system.ini!!!

[vedrancek]

Hello,

I have a XPS-D controller with the following information: Snapshot version: XPS-D-N13018 QNX kernel version: QNX kernel configuration V3.0.22 (20191107, SYSTEM, apic mode) Stage database revision: StageDataBase V4.3.6 - XPS Controller

When trying to connect to connect to it:

from newportxps import NewportXPS xps = NewportXPS('192.168.254.254', username='Administrator', password='Administrator') I get the following error message: Could not read system.ini!!!"

I have system.ini file on the controller and when connecting the controller with dedicated software that came with the whole system there is no problem. Also, everything from the web interface of the controller side looks fine. Also when trying to connect to it, there are no other open connections with the controller.

I am using Python 3.7.4.

Any ideas, suggestions? What did I miss? Please advise.

This is the system.ini file: `; ######################################################################################### ; # Generated from manual configuration by Administrator on Fri, 20 Dec 2019 18:07:40 GMT # ; #########################################################################################

[GENERAL] BootScriptFileName = SetGPIO3output1.tcl BootScriptArguments =

[GROUPS] SingleAxisInUse = Z, P SpindleInUse = XYInUse = XY XYZInUse = MultipleAxesInUse =

; ############################################# ; # Single axis group 'Z' and its positioner: Z

[Z] PositionerInUse = Z

[Z.Z] PlugNumber = 5 StageName = IMS@IMS100V@XPS-DRV11 SecondaryPositionerGantry = Disabled

; ############################################# ; # Single axis group 'P' and its positioner: P

[P] PositionerInUse = P

[P.P] PlugNumber = 4 StageName = AXIS_PZ SecondaryPositionerGantry = Disabled

; ############################################## ; # XY axes group 'XY' and its positioners: X, Y

[XY] PositionerInUse = X, Y InitializationAndHomeSearchSequence = Together XMappingFileName = XMappingLineNumber = 0 XMappingColumnNumber = 0 XMappingMaxPositionError = 0 YMappingFileName = YMappingLineNumber = 0 YMappingColumnNumber = 0 YMappingMaxPositionError = 0

[XY.X] PlugNumber = 1 StageName = AXIS_X SecondaryPositionerGantry = Disabled

[XY.Y] PlugNumber = 2 StageName = AXIS_Y SecondaryPositionerGantry = Disabled `

Thank you in advance!

Best regards, Vedrancek

[newville]

@vedrancek Sorry for the trouble. FWIW, I am using this code with an XPS-D (and C and Q!), so I think that we should be able to get it to work. Offhand, I have 3 thoughts:

a) can you provide the full traceback to see where read_systemini() is failing. b) can you verify that you are using sftp to connect to the XPS? (I suspect this is not the issue....) c) I don't think I've ever tested with an XY Group or XYZGroup, so this could be where read_systemini is failing to read/parse the system.ini file. I suspect this is the problem....

[newville]

@vedrancek a quick test suggests that read_systemini() ought to be able to parse and use that system.ini file, so I'm less sure what's going on...

[vedrancek]

@newville thank you for the quick response and sorry for my later one. I was out of the office. Yesterday I went through the code and for starters I wanted to know if all libraries and modules can be manually loaded from python prompt. When entering:

from XPS_C8_drivers import XPS, XPSException Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'XPS_C8_drivers'

Did I something forgotten to install? When installing your driver I ran setup.py

Also since I am at the beginning of my Phthon journey, could you please guide me, how to do a traceback to see where read_systemini() is failing?

Cheers, Vedrancek

[newville]

@vedraneck Well, after installing with python setup.py install or from the source folder directory, you should be able to do:

>>> from newportxps import NewportXPS
>>> from newportxps.XPS_C8_drivers import XPS, XPSException

Connecting to a running XPS could then be

>>> my_xps = NewportXPS('192.168.0.253')

or whatever IP address your XPS is using.

Alternatively, you could connect to the XPS using the lower-level interface:

>>> my_xps = XPS()
>>> ssid = my_xps.TCP_ConnectToServer('192.168.0.253', 5001, 10) 
>>> my_xps.Login('Administrator', 'AdminPassword')
>>> err, firmware_version = my_xps.FirmwareVersionGet(ssid)

and so forth.

[hududed]

Hi @newville I am facing the same issue.

I have attempted both remote (DHCP-assigned IP 192.168.254.254) and host connection (static 192.168.0.101 / subnet 255.255.255.0). I have also attempted both X,Y,Z as single axis, and XY- and Z-axes groups.

In both cases, system.ini could not be read.

Any suggestions? How do I run full trace on read_systemini()?

[newville]

@hududed When you say that you are saying that you are "facing the issue" to an issue that is six months old, you force anyone who might try to help you to read the older issue, see what that was about, what any resolution might have been and how it might apply to the new report.

So rather than spend time trying to understand your trouble, I went back to try to understand the previous problem. As it turns out, I don't know what the solution to the earlier report really was. I made some suggestions but never heard back what the problem was.

Do want to focus on how this is like that unresolved issue? You include no code and no traceback or any indication that you tried any of the suggestions made for the apparently similar problem.

[hududed]

I apologize.

I ran the following:

from newportxps import NewportXPS
from newportxps.XPS_C8_drivers import XPS, XPSException

xps = NewportXPS('192.168.254.254')

and got the first error: Could not read system.ini!!!

I ran print(xps.status_report()) but no the groups of were detected:

Could not read system.ini!!!
Could not read system.ini!!!
Could not read system.ini!!!
Could not read system.ini!!!
# XPS host:         192.168.254.254 (XPS-1B81)
# Firmware:         XPS-D-N13019
# Current Time:     Wed Jul 29 15:32:15 2020
# Last Reboot:      Wed Jul 29 14:21:26 2020
# Trajectory Group: None
# Groups and Stages

I ran just these codes above with the device connected as remote, as host, and manual configuration (pg. 7 of this) of X Y and Z axes as single axes, as well as XY-grouped and Z axes - as you suspected from the previous issue above (see also my system.ini ).

As to the traceback, I am not sure how to provide this and need your advise.

My system.ini file:

; #########################################################################################
; # Generated from manual configuration by Administrator on Wed, 29 Jul 2020 21:35:09 GMT #
; #########################################################################################

[GENERAL]
BootScriptFileName = 
BootScriptArguments = 

[GROUPS]
SingleAxisInUse = Z, X, Y
SpindleInUse = 
XYInUse = 
XYZInUse = 
MultipleAxesInUse = 

; #################################################
; # Single axis group 'Z' and its positioner: Z-MFA

[Z]
PositionerInUse = Z-MFA

[Z.Z-MFA]
PlugNumber = 3
StageName = MFA@MFA-CC@XPS-DRV11
SecondaryPositionerGantry = Disabled

; #################################################
; # Single axis group 'X' and its positioner: X-MFA

[X]
PositionerInUse = X-MFA

[X.X-MFA]
PlugNumber = 1
StageName = MFA@MFA-CC@XPS-DRV11
SecondaryPositionerGantry = Disabled

; #################################################
; # Single axis group 'Y' and its positioner: Y-MFA

[Y]
PositionerInUse = Y-MFA

[Y.Y-MFA]
PlugNumber = 2
StageName = MFA@MFA-CC@XPS-DRV11
SecondaryPositionerGantry = Disabled

[newville]

@hududed Hm, not sure. I guess we should comment out the bare "try / except" in connect() to just be:

        err, val = self._xps.FirmwareVersionGet(self._sid)
        self.firmware_version = val
        self.ftphome = ''

        if 'XPS-D' in self.firmware_version:
            err, val = self._xps.Send(self._sid, 'InstallerVersionGet(char *)')
            self.firmware_version = val
            self.ftpconn = SFTPWrapper(**self.ftpargs)
        else:
            self.ftpconn = FTPWrapper(**self.ftpargs)
            if 'XPS-C' in self.firmware_version:
                self.ftphome = '/Admin'
        #### try:
        self.read_systemini()
        #### except:
        ####    print("Could not read system.ini!!!")

I would suggest making that change and trying again - that should at least try to expose the problem better.
Off-hand, I'm not sure what the problem is. The config file looks ok, so I would guess it is the sftp connection.

[hududed]

I think you're right. Removing the try/except code removes that error.

Continuing from before, if I run xps.read_systemini() I get the error that pysftp is not installed. After installing that and rerunning read_systemini() I get:

SSHException                              Traceback (most recent call last)
<ipython-input-3-49181f127cb3> in <module>
----> 1 xps.read_systemini()

~\Anaconda3\envs\controller2\lib\site-packages\newportxps-0.1-py3.7.egg\newportxps\newportxps.py in read_systemini(self)
    161         this is part of the connection process
    162         """
--> 163         self.ftpconn.connect(**self.ftpargs)
    164         self.ftpconn.cwd(os.path.join(self.ftphome, 'Config'))
    165         lines = self.ftpconn.getlines('system.ini')

~\Anaconda3\envs\controller2\lib\site-packages\newportxps-0.1-py3.7.egg\newportxps\ftp_wrapper.py in connect(self, host, username, password)
     73         self._conn = pysftp.Connection(self.host,
     74                                        username=self.username,
---> 75                                        password=self.username)
     76 
     77     def save(self, remotefile, localfile):

~\Anaconda3\envs\controller2\lib\site-packages\pysftp\__init__.py in __init__(self, host, username, private_key, password, port, private_key_pass, ciphers, log, cnopts, default_path)
    130         # check that we have a hostkey to verify
    131         if self._cnopts.hostkeys is not None:
--> 132             self._tconnect['hostkey'] = self._cnopts.get_hostkey(host)
    133 
    134         self._sftp_live = False

~\Anaconda3\envs\controller2\lib\site-packages\pysftp\__init__.py in get_hostkey(self, host)
     69         kval = self.hostkeys.lookup(host)  # None|{keytype: PKey}
     70         if kval is None:
---> 71             raise SSHException("No hostkey for host %s found." % host)
     72         # return the pkey from the dict
     73         return list(kval.values())[0]

SSHException: No hostkey for host 192.168.254.254 found.

I am not too familiar with sftp and hostkeys, I am inclined to set this off (although I am not sure how to do that) but this maybe a security-risk. Is the hostkey something delivered with the XPS-D instrument? Or something we set ourselves? Or was this code written with an older version of pysftp without hostkey checking?

[newville]

@hududed Hm, I'm not sure - I don't see that problem, and I'm pretty sure I don't have hostkeys set for the XPS-D. I think I do not have anything special set up for sftp in the .ssh account for the user account I typically use or in /etc/ssh. I can say that I only connect to an XPS-D that happens to have real IP addresses assigned on our subnet, but we're working to migrate those to a private network sometime in the next couple of months.

Please let us know if you figure something out that works!

[hududed]

Hello @newville

after some debugging I am more certain it has to do with the connection and pysftp.

    def read_systemini(self):
        """read group info from system.ini
        this is part of the connection process
        """
        self.ftpconn.connect(**self.ftpargs)

The code stops at the final line here, which steps into ftp_wrapper.py it will try to import pysftp and pass:

HAS_PYSFTP = False
try:
    import pysftp
    HAS_PYSFTP = True
except ImportError:
    pass

then the connection would not work but the ValueError is not raised:

    def connect(self, host=None, username=None, password=None):
        if host is not None:
            self.host = host
        if username is not None:
            self.username = username
        if password is not None:
            self.password = password

        if not HAS_PYSFTP:
            raise ValueError("pysftp not installed.")

        self._conn = pysftp.Connection(self.host,
                                       username=self.username,
                                       password=self.username)

Could you check which version of the pysftp you have installed? As it appears that the newer versions need to check for the SSH hostkey.

[newville]

@hududed I'm using pysftp 0.2.9 and Python 3.7.4 (from Anaconda Python) on Centos7.

I think the good news is that you can just debug that to figure out what is going on with the pysftp.Connection().

This all sort of worked for me, so I have not investigated in detail. But I do see this:

>>> import pysftp
>>> cnopts = pysftp.CnOpts()
>>> cnopts.get_hostkey('foo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xas_user/anaconda3/lib/python3.7/site-packages/pysftp/__init__.py", line 71, in get_hostkey
    raise SSHException("No hostkey for host %s found." % host)
paramiko.ssh_exception.SSHException: No hostkey for host foo found.

>>> cnopts.get_hostkey(xpsd_ipaddress) 
<paramiko.ecdsakey.ECDSAKey object at 0x7f234619b410>

So, I would guess that is probably what you need to get too. The XPS-D I connect to is listed in my .ssh/known_hosts file with an entry like

164.XXX.YYY.ZZZ ecdsa-sha2-nistp256 AAAAE2VjZHNh...

I do not recall adding that by hand, but I probably did a test using the command-line sftp to this machine early on.

I hope we can come up with a better "user guide" on this topic, once we figure it out!

[hududed]

Oh! I have an intermediate solution by disabling the hostkey check - but ofc this has security risks.

# XPS host:         192.168.254.254 (XPS-1B81)
# Firmware:         XPS-D-N13019
# Current Time:     Thu Jul 30 09:27:30 2020
# Last Reboot:      Wed Jul 29 14:35:05 2020
# Trajectory Group: None
# Groups and Stages
Z (singleaxisinuse), Status: Ready state from homing
   Z.Z-MFA (MFA@MFA-CC@XPS-DRV11)
      Hardware Status: First driver powered on - ZM low level
      Positioner Errors: OK
X (singleaxisinuse), Status: Ready state from homing
   X.X-MFA (MFA@MFA-CC@XPS-DRV11)
      Hardware Status: First driver powered on - ZM low level
      Positioner Errors: OK
Y (singleaxisinuse), Status: Ready state from homing
   Y.Y-MFA (MFA@MFA-CC@XPS-DRV11)
      Hardware Status: First driver powered on - ZM low level
      Positioner Errors: OK

I made the PR for the intermediate solution anyway.

So, I guess the question is how does one attain this xpsd_ipaddress ?

[newville]

@hududed Oh, for me that is just the real IP address of the XPS-D I'm using, assigned with our DHCP server.

[vedrancek]

Hello all,

long time no hear from me. I had to focus on another project with the system so I had to put this aside. I still want to try to run the driver from Python so as soon as I will be finished I will go back and try to make it work. Hopefully, this will be in the next 3-4 months.

I will be back with updates on my progress. So please don't close this thread.

Thanks and regards, Vedrancek

[newville]

@vedrancek Well, I think I never really understood the problem you were having. @hududed sort of took over this conversation, with an issue that is maybe related.

From my perspective, revisiting old conversations sort of means that you expect me to remember the details of the unresolved issue. And, like, I'm not going to remember. And for sure, give a good report. If you don't include an actual script and a traceback that will be the first thing I ask about.

[hududed]

@newville

if need be we I can open new issue - but I thought there's overlap here.

With regards to setting the SSH, I am using puttyGen to create the keys, but from what I read I have to place the public key in ~/.ssh/authorized_keys in the xps instrument. However, I dont have permissions when connected to the instrument. Do you recall having to do this?

[newville]

@hududed I think that ~/.ssh/authorized_keys on the client machine trying to connect to the XPS-D is not relevant here. Or, are you trying to set that on the XPS-D? I have never had to do any such thing.
On the workstation from which I sftp to my XPS-D and/or use this python module, I do not have an ~/.ssh/authorized_keys file. But that would be to list keys that could connect to the workstation.

[hududed]

I have decided to skip this ssh problem, since the read_systemini works and I can control the motors for now.

[newville]

@hududed OK, did you ever resolve the problem with authorized_keys? I'm still sort of wondering how we should best handle that problem for other folks who run into it.

[hududed]

@newville unfortunately I didn't, but this may be due to my inexperience with assigning SSH public key specifically to the instrument.

I used these steps which did not work (see the steps in Copy Public Key To Server):

1. Generated ecdsa-sha2-nistp256 key pair using PuTTYGen.
2. Saved the private key in ~/.ssh.
3. Copied the text ecdsa-sha2-nistp256 AAAAB3NzaC1yc2EAAAABJ.... and saved as public key file in ~/.ssh/known_hosts/xps_public.pub.
4. Edited the public key file by adding IP 192.168.XXX.XXX ecdsa-sha2-nistp256 AAAAB3N....
5. Ran the code:

   
   >>>import pysftp
   >>>cnopts = pysftp.CnOpts()
   >>>cnopts.hostkeys.load('~/.ssh/known_hosts/xps_public.pub')

cnopts.get_hostkey(xpsd_ipaddress) <paramiko.ecdsakey.ECDSAKey object at 0x2f434319b410>

xps = NewportXPS(xpsd_ipaddress) Could not read system.ini!!!

[lockhart]

fwiw I'm seeing the exact same symptoms after installing newportxps from the repo using conda pip. The first symptom was "Could not read system.ini!!!" with no other clues. After installing pysftp the error gets more specific as indicated in the thread:

ERROR: sftp connection to 198.202.125.168 failed
You may need to add the host keys for your XPS to your
ssh known_hosts file, using a command like this:
  ssh-keyscan 198.202.125.168 >> ~/.ssh/known_hosts
Exception ignored in: <function Connection.__del__ at 0x7f09201df8c0>
Traceback (most recent call last):
  File "/proj/parvi/release/anaconda3/envs/parvi/lib/python3.7/site-packages/pysftp/__init__.py", line 1013, in __del__
    self.close()
  File "/proj/parvi/release/anaconda3/envs/parvi/lib/python3.7/site-packages/pysftp/__init__.py", line 784, in close
    if self._sftp_live:
AttributeError: 'Connection' object has no attribute '_sftp_live'
Could not read system.ini!!!

I'll pursue the other clues in this thread to see if I can get past this. (The "_sftp_live" issue would seem to be a separate small problem). Thanks!

(edit) Uh, actually following the suggestion in the error message Worked For Me:

ssh-keyscan 198.202.125.168 >> ~/.ssh/known_hosts

Thanks for the thread!

[newville]

@lockhart OK, that sounds like several people have seen this and agree on a fix. The current master version is tagged as v0.2 and should have better error messages when pysftp fails to connect in this way.

[hududed]

Not sure if this is relevant here, but somehow my PC has a specific error with pysftp. I updated newportxps to v0.2, although its the same error with v0.1 now .. When initializing NewportXPS(host) it constantly gives the error Could not read system.ini!!!

Looking further, it seems pysftp never finds any keys for me now. I previously overcome this by disabling the cnopts.hostkeys = None but now initializing pysftp.CnOpts() gives the error:

import pysftp
pysftp.CnOpts()
cnopts.hostkeys = None
---------------------------------------------------------------------------
HostKeysException                         Traceback (most recent call last)
<ipython-input-5-6cfda8b75e40> in <module>
      1 import pysftp
----> 2 pysftp.CnOpts()

~\miniconda3\envs\pycontrol\lib\site-packages\pysftp\__init__.py in __init__(self, knownhosts)
     62         else:
     63             if len(self.hostkeys.items()) == 0:
---> 64                 raise HostKeysException('No Host Keys Found')
     65 
     66     def get_hostkey(self, host):

HostKeysException: No Host Keys Found

Has anyone experienced this ? Or does anyone know where the official pysftp repo is so I can raise the issue there

[newville]

@hududed that looks very much like the same real problem: "No Host Keys Found" is sftp telling you that it cannot find host keys. To communicate with the XPS-D, you do need to be able to connect to it with sftp.

[hududed]

import pysftp cnopts = pysftp.CnOpts() cnopts.hostkeys.load('~/.ssh/known_hosts/xps_public.pub')

cnopts.get_hostkey(xpsd_ipaddress) <paramiko.ecdsakey.ECDSAKey object at 0x2f434319b410>

xps = NewportXPS(xpsd_ipaddress) Could not read system.ini!!!

@newville This is rather different, from this quote from Aug 3 above - I was able to run pysftp.CnOpts(). Anyway I have raised the issue https://github.com/Clean-Cole/pysftp/issues/3 which seems to be the official pysftp github repo

[newville]

@hududed yeah, sorry but I just don't think I can really offer any advice other than "get sftp" to work.
I think this does not have anything to do with Python. I recommend trying just the command-line utility: sftp Administrator@XPS_IPDARRRESS

If you have done ssh_keyscan on the XPS-D host and you still get the same error, then I would suggest asking your IT folks for help.

I think this is really not an issue with the newportxps python module, except insofar as it wasn't giving a very informative message about why it could not read the system.ini file, and that is now fixed.

[hududed]

I have found the issue.

ssh-keyscan 192.168.254.254 > ~/.ssh/known_hosts somehow saved the files in UTF-16 encoding. For those having this problem on windows - I used notepad++ > Encoding > UTF-8 for conversion, then pysftp.CnOpts() gives no No Host Keys error.

[openindus]

Hello,

I'm facing a similar issue while attempting to read the system.ini file. The XPS seems to refuse the connexion.

I've tried both host port and remote port without any success. By reading all the previous post, i've also tested the different possibilities adding the host to the file known_hosts or by disabling the hostkey check.

I can access to the XPS with Filezilla configured with SFTP protocol. ssh command line is also working. Windows firewall was desactivated but it ends the same.

My test script is the following :

from newportxps import NewportXPS xps = NewportXPS('192.168.2.103', username='Administrator', password='Administrator') print(xps.read_systemini())

A capture showing all versions installed :

An other one which describe the configuration of the newportxps controller :

Finally, trace of the error when the function read_systemini() is called :

I can also share the system.ini file but don't think it is neccessary as the problem is the ftp access to the xps.

[newville]

@openindus Sorry for the trouble. But I think that as with previous cases here, the issue is probably really about getting your workstation/ computer to connect with sftp to the Newport XPS-D controller.

Well, I'm assuming that you are trying to connect to an XPS-D, but am not certain of that. It seems odd that your traceback looks like it is using the ftplib. Can you verify whether this is an XPS-D or an older "Q" or "C" model? Maybe the problem is in our "identify the firmware model of this XPS" code.

[openindus]

Thanks for this remark. It's exactly the problem. Firmware version returns the string HXP-D Standard #50000 but the if condition test if XPS-D' in self.firmware_version: is not good. I just changed XPS-D by HXP-D and it's working fine now.

[newville]

@openindus Ah, OK. That must be some new variation of a controller - I see now that's a hexapod controller, possibly with some different functionality -- and you can get a little game controller to run it... that's kind of cool.

I have not looked any further, but are there any other features to expose?

If changing the "firmware version" to match "HXP-D" or "XPS-D" works, then I can update the code for the next person who wants to use one of these.

[openindus]

Hello, The current code does not have all the functions to use the hexapod. Do you have the possibility to provide us with a python code allowing to have these functions ?

[newville]

@openindus I don't know what the new hexapod functions are. Do you know whether there is an update to XPS_C8_drivers.py, with these? And do you know what "higher level" wrapper you would like around these XPS functions? Are you willing to write some of those wrappings?

[maxawake]

I am using the HXP50-MECA Hexapod with the HXP50-ELEC-D controller. I ran into the same problem Could not read system.ini!!! without any clues about what causes the issue. Upon printing the exception i realized i forgot to install pysftp. After installation i get the Host key error. I was able to solve it by adding this to the ftp_wrapper.py ( see this stackoverflow question https://stackoverflow.com/questions/65002585/connection-object-has-no-attribute-sftp-live-when-pysftp-connection-fails )

cnopts = pysftp.CnOpts()
cnopts.hostkeys = None
self._conn = pysftp.Connection(self.host,
    username=self.username,
    password=self.password,
    cnopts=cnopts)

They say this might be a bug in pysftp but as long as its not fixed on their side i think it should be circumvented in the newportxps module.

However, with this small change everything works like a charm! Thank you so much for this!!

[newville]

@maxawake hm, weird but believable. Do you think it would be reasonable to do something like this in SFTPWrapper.connect()::

try:
    self._conn = pysftp.Connection(self.host, username=self.username, password=self.password)
except:  # should this be a specific exception about Hostkeys?
    try:
        cnopts = pysftp.CnOpts()
        cnopts.hostkeys = None
        self._conn = pysftp.Connection(self.host, username=self.username, password=self.password, 
                                       cnopts=cnopts)
   except:  # ok, really failed
       print(f"ERROR sftp connection to {self.host} failed") 
       print(helpful_hints_for_sftp)