Code completion for Pandas is not working

[PhilipYip1988]

Issue Report Checklist

• [x] Searched the issues page for similar reports
• [x] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• [x] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• [x] Could not reproduce inside jupyter qtconsole (if console-related)
• [x] Tried basic troubleshooting (if a bug/error)
  • [x] Restarted Spyder
  • [x] Reset preferences with spyder --reset
  • [x] Reinstalled the latest version of Anaconda
  • [x] Tried the other applicable steps from the Troubleshooting Guide
• [x] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

pandas identifiers for Series and DataFrame don't display properly in the script editor. docstring for a class in a standard module does not display when module and class name are the same e.g. datetime.datetime or time.time

What steps reproduce the problem?

1. Create Script file with a pandas dataframe. For example:

import numpy as np
import pandas as pd
x = np.array([0, 1, 2, 3, 4, 5])
y = 2 * x
df = pd.DataFrame({'x': x, 'y': y})
import datetime
import time

2. Then examine identifiers or docstring in the script editor in response to:

a. np. # display correctly b. x. # display correctly c. pd. # display correctly d. df. # display correctly Spyder 5, don't display Spyder 6 e. pd.Series. # display correctly Spyder 5, don't display Spyder 6 f. pd.DataFrame. # display correctly Spyder 5, don't display Spyder 6 g. df.x. # Series accessed from DataFrame as an attribute, don't display Spyder 5 or 6 h. df['x']. # Series accessed from DataFrame as an index, don't display Spyder 5 or 6 i. datetime. # Identifiers display correctly Spyder 5 or 6 j. datetime.timedelta # Identifiers display correctly Spyder 5 or 6 k. datetime.timedelta() # Docstring displays correctly Spyder 5 or 6 l. datetime.datetime. # identifiers display correctly Spyder 5 or 6 m. datetime.datetime() # Docstring does not display Spyder 5 or 6 n. time. # identifiers display correctly Spyder 5 or 6 o. time.sleep() # Docstring displays correctly Spyder 5 or 6 p. time.time() # Docstring does not display Spyder 5 or 6 q. time.time_ns() # Docstring does not display Spyder 5 or 6

What is the expected output? What do you see instead?

d, e and f should work in Spyder 6 like Spyder 5.

g and h should display identifiers of e.

m, p and q should display docstring.

Paste Traceback/Error Below (if applicable)

PASTE TRACEBACK HERE

Versions

Spyder 5

• Spyder version: 5.4.3 (conda)
• Python version: 3.11.5 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.10
• Operating System: Windows 10

Spyder 6

• Spyder version: 6.0.0a3 (conda)
• Python version: 3.11.7 64-bit
• Qt version: 5.15.8
• PyQt5 version: 5.15.9
• Operating System: Windows-10-10.0.22631-SP0

Dependencies

Spyder 5

# Mandatory:
atomicwrites >=1.2.0                             :  1.4.0 (OK)
chardet >=2.0.0                                  :  4.0.0 (OK)
cloudpickle >=0.5.0                              :  2.2.1 (OK)
cookiecutter >=1.6.0                             :  2.5.0 (OK)
diff_match_patch >=20181111                      :  20200713 (OK)
intervaltree >=3.0.2                             :  3.1.0 (OK)
IPython >=7.31.1,<9.0.0,!=8.8.0,!=8.9.0,!=8.10.0 :  8.15.0 (OK)
jedi >=0.17.2,<0.19.0                            :  0.18.1 (OK)
jellyfish >=0.7                                  :  1.0.1 (OK)
jsonschema >=3.2.0                               :  4.19.2 (OK)
keyring >=17.0.0                                 :  23.13.1 (OK)
nbconvert >=4.0                                  :  7.10.0 (OK)
numpydoc >=0.6.0                                 :  1.5.0 (OK)
paramiko >=2.4.0                                 :  2.8.1 (OK)
parso >=0.7.0,<0.9.0                             :  0.8.3 (OK)
pexpect >=4.4.0                                  :  4.8.0 (OK)
pickleshare >=0.4                                :  0.7.5 (OK)
psutil >=5.3                                     :  5.9.0 (OK)
pygments >=2.0                                   :  2.15.1 (OK)
pylint >=2.5.0,<3.0                              :  2.16.2 (OK)
pylint_venv >=2.1.1                              :  2.3.0 (OK)
pyls_spyder >=0.4.0                              :  0.4.0 (OK)
pylsp >=1.7.2,<1.8.0                             :  1.7.2 (OK)
pylsp_black >=1.2.0                              :  1.2.1 (OK)
qdarkstyle >=3.0.2,<3.2.0                        :  3.0.2 (OK)
qstylizer >=0.2.2                                :  0.2.2 (OK)
qtawesome >=1.2.1                                :  1.2.2 (OK)
qtconsole >=5.4.2,<5.5.0                         :  5.4.2 (OK)
qtpy >=2.1.0                                     :  2.4.1 (OK)
rtree >=0.9.7                                    :  1.0.1 (OK)
setuptools >=49.6.0                              :  68.0.0 (OK)
sphinx >=0.6.6                                   :  5.0.2 (OK)
spyder_kernels >=2.4.3,<2.5.0                    :  2.4.4 (OK)
textdistance >=4.2.0                             :  4.2.1 (OK)
three_merge >=0.1.1                              :  0.1.1 (OK)
watchdog >=0.10.3                                :  2.1.6 (OK)
zmq >=22.1.0                                     :  25.1.0 (OK)

# Optional:
cython >=0.21                                    :  None (NOK)
matplotlib >=3.0.0                               :  3.8.0 (OK)
numpy >=1.7                                      :  1.26.2 (OK)
pandas >=1.1.1                                   :  2.1.1 (OK)
scipy >=0.17.0                                   :  1.11.4 (OK)
sympy >=0.7.3                                    :  1.12 (OK)

Spyder 6

# Mandatory:
atomicwrites >=1.2.0              :  1.4.1 (OK)
chardet >=2.0.0                   :  5.2.0 (OK)
cloudpickle >=0.5.0               :  3.0.0 (OK)
cookiecutter >=1.6.0              :  2.5.0 (OK)
diff_match_patch >=20181111       :  20230430 (OK)
intervaltree >=3.0.2              :  3.1.0 (OK)
IPython >=8.13.0,<9.0.0,!=8.17.1  :  8.18.1 (OK)
jedi >=0.17.2,<0.20.0             :  0.19.1 (OK)
jellyfish >=0.7                   :  1.0.3 (OK)
jsonschema >=3.2.0                :  4.20.0 (OK)
keyring >=17.0.0                  :  24.3.0 (OK)
nbconvert >=4.0                   :  7.13.1 (OK)
numpydoc >=0.6.0                  :  1.6.0 (OK)
paramiko >=2.4.0                  :  3.4.0 (OK)
parso >=0.7.0,<0.9.0              :  0.8.3 (OK)
pexpect >=4.4.0                   :  4.8.0 (OK)
pickleshare >=0.4                 :  0.7.5 (OK)
psutil >=5.3                      :  5.9.7 (OK)
pygments >=2.0                    :  2.17.2 (OK)
pylint >=2.5.0,<3.1               :  2.17.7 (OK)
pylint_venv >=3.0.2               :  3.0.3 (OK)
pyls_spyder >=0.4.0               :  0.4.0 (OK)
pylsp >=1.9.0,<1.10.0             :  1.9.0 (OK)
pylsp_black >=1.2.0,<3.0.0        :  2.0.0 (OK)
pyuca >=1.2                       :  1.2 (OK)
qdarkstyle >=3.2.0,<3.3.0         :  3.2.3 (OK)
qstylizer >=0.2.2                 :  0.2.2 (OK)
qtawesome >=1.3.0                 :  1.3.0 (OK)
qtconsole >=5.5.0,<5.6.0          :  5.5.1 (OK)
qtpy >=2.4.0                      :  2.4.1 (OK)
rtree >=0.9.7                     :  1.1.0 (OK)
setuptools >=49.6.0               :  68.2.2 (OK)
sphinx >=0.6.6                    :  7.2.6 (OK)
spyder_kernels >=3.0.0b3,<3.0.0b4 :  3.0.0b3 (OK)
superqt >=0.6.1,<1.0.0            :  0.6.1 (OK)
textdistance >=4.2.0              :  4.5.0 (OK)
three_merge >=0.1.1               :  0.1.1 (OK)
watchdog >=0.10.3                 :  3.0.0 (OK)
zmq >=22.1.0                      :  25.1.2 (OK)

# Optional:
cython >=0.21                     :  3.0.7 (OK)
matplotlib >=3.0.0                :  3.8.2 (OK)
numpy >=1.7                       :  1.26.2 (OK)
pandas >=1.1.1                    :  2.1.4 (OK)
scipy >=0.17.0                    :  1.11.4 (OK)
sympy >=0.7.3                     :  1.12 (OK)

[ccordoba12]

Hey @PhilipYip1988, thanks for reporting. I can't reproduce this on Linux.

@dalthviz, can you try it on Windows? Thanks!

[dalthviz]

Although I'm not sure if I'm testing this correctly (when you say docstrings and identifiers @PhilipYip1988 you mean like the completion options that are retrieved and also that for the highlighted completion option its documentation is shown, right?). Checking locally I got the same behavior regardless of testing with Spyder 6 from conda and Spyder 5.5.0 for all the cases. However, indeed there are cases when Spyder does not provide any docstring/completion from the Editor like the ones mentioned over the OP, maybe some regression regarding completion happened between Spyder 5.4.3 and 5.5.0?

[PhilipYip1988]

Here are some screenshots, no identifiers show for pd:

Like they do for numpy:

Also for the docstring, datetime.time shows:

Also for the docstring, datetime.datetime does not show:

[xieyiqun884886]

I would like to sincerely ask whether there is a solution for this.

[ccordoba12]

We don't know why this is happening, sorry.

[PhilipYip1988]

Retested with Spyder 6.0.0a5.

datetime

• datetime. # works as expected


• datetime.timedelta # works as expected


• datetime.timedelta() # works as expected


• datetime.datetime. # works as expected


• datetime.datetime() # Docstring does not display when input with open parenthesis

time

• time. # works as expected


• time.sleep # works as expected


• time.sleep() # works as expected


• time.time # works as expected


• time.time() # works as expected


• time.time_ns # works as expected


• time.time_ns() # Docstring does not display when input with open parenthesis

numpy

• np. # works as expected


• np.array # works as expected


• np.array() # works as expected


• x. # works as expected


• x.sum # works as expected


• x.sum() #works as expected


pandas

• pd. # works as expected


• pd.Series # works as expected


• pd.Series() # works as expected


• pd.Series. # no Series identifiers display

• pd.DataFrame # works as expected


• pd.DataFrame() # works as expected


• pd.DataFrame. # no DataFrame identifiers display

• df. # no DataFrame identifiers display

• df.x. # no Series identifiers display (attribute access)

• df['x']. # no Series identifiers display (key access)

[ccordoba12]

@PhilipYip1988, thanks for the very detailed update! I collapsed the screenshots of those things that are working and left the ones that are not, so we can focus on solving them.

[ccordoba12]

@dalthviz, please take care of this one.

[dalthviz]

Note: Seems like the majority of cases listed here are not working as expected due to a change over the upper limit of jedi done at https://github.com/spyder-ide/spyder/commit/60ae3c650d2c5778a12f5b5a3fcfec8fef63f507 (so it affects Spyder >= 5.5.0).

When checking things with jedi 0.18.2 and latest master things work except for:

• Cases like df.x. and df['x']: From the Console seems like this works, which thinking about it, seems like something that we have discussed previously and would imply to somehow connect the completions from the Console to the ones from the Editor (to have such completions you would need to actually execute the code, right @ccordoba12 ?)

• Cases like time.time_ns(): I think the docstring/signature tooltip is not being displayed when you input parenthesis since those methods/functions don't have arguments in their signature. Basically, the current way Spyder handles this is by showing the tooltip when typing the parenthesis only if the function/method has arguments. Still, if you want to see the tooltip you can hover over the definition and it should appear regardless of the function/method having args/kwargs or not. As an example of the handling of something with args/kwargs vs something without them: Should we change this behavior @ccordoba12 ?

[ccordoba12]

Note: Seems like the majority of cases listed here are not working as expected due to a change over the upper limit of jedi

Thanks for digging into this @dalthviz! Let's leave it for 6.0.1 then because it's not so simple to fix (you'll need to run git bisect on the Jedi repo between 0.18.2 and the latest release to find exactly what introduced the problem and report that upstream).

Cases like df.x. and df['x']: From the Console seems like this works, which thinking about it, seems like something that we have discussed previously and would imply to somehow connect the completions from the Console to the ones from the Editor (to have such completions you would need to actually execute the code, right @ccordoba12 ?)

Correct, that's the only way to fix that. But that will work only if you've run your code first.

Should we change this behavior @ccordoba12 ?

Yeah, I think that's a good idea and you can implement it for 6.0.0 because it doesn't sound too hard to fix.