jakebailey
commented
5 years ago This is #1576. Are you on the latest daily build?
Closed malmaud closed 5 years ago
jakebailey
commented
5 years ago This is #1576. Are you on the latest daily build?
jakebailey
commented
5 years ago Sorry, this is not #1576, but related.
jakebailey
commented
5 years ago I do see your version as 0.4.64 from that file path, 0.4.66's fix may impact this, so I'd appreciate it if you could enable the daily build and see if the same thing happens.
jakebailey
commented
5 years ago I have no idea why this error is coming through the document symbol call, but this to me looks like we're trying to scrape one of our own files (which is really, really weird).
@malmaud Do you have full logs including the interpreter paths we normally print?
malmaud
commented
5 years ago Here's the complete log (still happening on the daily build):
https://gist.github.com/malmaud/6b83cfeb864ddbf927888cece2f79152
MikhailArkhipov
commented
5 years ago This is the problem
[Info - 2:43:09 PM] Interpreter search paths:
[Info - 2:43:09 PM] /Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67Extension folder is on Python search path
MikhailArkhipov
commented
5 years ago However, it would be good for the indexer to limit to Python files.
jakebailey
commented
5 years ago The thing is that the indexer should be explicitly ignoring anything in an interpreter path, and the function which throws this exception (FromFullPath) is only used in the analysis.
malmaud
commented
5 years ago That folder isn't in sys.path when I just run python from the command line, so if that folder is being added, it must be being added by the vscode python extension or language server.
jakebailey
commented
5 years ago Can you set "python.trace.server": "verbose", reload, and get a copy of the first few messages?
The output is going to be very large, so I'm only interested in things up to the "Interpreter search paths" message.
jakebailey
commented
5 years ago Actually, that may contain stuff you don't want to share (your configuration), so if you could search for both the initialize request, and the workspace/configuration request that occurs near the start, that'd be what we're interested in.
Trying to see who is adding the bad path.
malmaud
commented
5 years ago There's nothing confidential, so I'm just including the whole first part of that output:
Trace - 3:43:38 PM] Sending request 'initialize - (0)'.
Params: {
"processId": 38937,
"rootPath": "/Users/malmaud/dropbox/code/qa",
"rootUri": "file:///Users/malmaud/dropbox/code/qa",
"capabilities": {
"workspace": {
"applyEdit": true,
"workspaceEdit": {
"documentChanges": true,
"resourceOperations": [
"create",
"rename",
"delete"
],
"failureHandling": "textOnlyTransactional"
},
"didChangeConfiguration": {
"dynamicRegistration": true
},
"didChangeWatchedFiles": {
"dynamicRegistration": true
},
"symbol": {
"dynamicRegistration": true,
"symbolKind": {
"valueSet": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26
]
}
},
"executeCommand": {
"dynamicRegistration": true
},
"configuration": true,
"workspaceFolders": true
},
"textDocument": {
"publishDiagnostics": {
"relatedInformation": true
},
"synchronization": {
"dynamicRegistration": true,
"willSave": true,
"willSaveWaitUntil": true,
"didSave": true
},
"completion": {
"dynamicRegistration": true,
"contextSupport": true,
"completionItem": {
"snippetSupport": true,
"commitCharactersSupport": true,
"documentationFormat": [
"markdown",
"plaintext"
],
"deprecatedSupport": true,
"preselectSupport": true
},
"completionItemKind": {
"valueSet": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25
]
}
},
"hover": {
"dynamicRegistration": true,
"contentFormat": [
"markdown",
"plaintext"
]
},
"signatureHelp": {
"dynamicRegistration": true,
"signatureInformation": {
"documentationFormat": [
"markdown",
"plaintext"
],
"parameterInformation": {
"labelOffsetSupport": true
}
}
},
"definition": {
"dynamicRegistration": true,
"linkSupport": true
},
"references": {
"dynamicRegistration": true
},
"documentHighlight": {
"dynamicRegistration": true
},
"documentSymbol": {
"dynamicRegistration": true,
"symbolKind": {
"valueSet": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
10,
11,
12,
13,
14,
15,
16,
17,
18,
19,
20,
21,
22,
23,
24,
25,
26
]
},
"hierarchicalDocumentSymbolSupport": true
},
"codeAction": {
"dynamicRegistration": true,
"codeActionLiteralSupport": {
"codeActionKind": {
"valueSet": [
"",
"quickfix",
"refactor",
"refactor.extract",
"refactor.inline",
"refactor.rewrite",
"source",
"source.organizeImports"
]
}
}
},
"codeLens": {
"dynamicRegistration": true
},
"formatting": {
"dynamicRegistration": true
},
"rangeFormatting": {
"dynamicRegistration": true
},
"onTypeFormatting": {
"dynamicRegistration": true
},
"rename": {
"dynamicRegistration": true,
"prepareSupport": true
},
"documentLink": {
"dynamicRegistration": true
},
"typeDefinition": {
"dynamicRegistration": true,
"linkSupport": true
},
"implementation": {
"dynamicRegistration": true,
"linkSupport": true
},
"colorProvider": {
"dynamicRegistration": true
},
"foldingRange": {
"dynamicRegistration": true,
"rangeLimit": 5000,
"lineFoldingOnly": true
},
"declaration": {
"dynamicRegistration": true,
"linkSupport": true
}
}
},
"initializationOptions": {
"interpreter": {
"properties": {
"InterpreterPath": "/Users/malmaud/anaconda3/bin/python",
"Version": "3.7.2",
"DatabasePath": "/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67"
}
},
"displayOptions": {
"preferredFormat": "markdown",
"trimDocumentationLines": false,
"maxDocumentationLineLength": 0,
"trimDocumentationText": false,
"maxDocumentationTextLength": 0
},
"searchPaths": [],
"typeStubSearchPaths": [
"/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67/Typeshed"
],
"cacheFolderPath": null,
"excludeFiles": [
"**/Lib/**",
"**/site-packages/**",
"**/node_modules",
"**/bower_components",
"**/.git",
"**/.svn",
"**/.hg",
"**/CVS",
"**/.DS_Store",
"**/__pycache__",
"**/.idea",
"**/.ipynb_checkpoints",
"**/.mypy_cache",
"**/.pytest_cache",
"**/.git/objects/**",
"**/.git/subtree-cache/**",
"**/node_modules/**",
".vscode/*.py",
"**/site-packages/**/*.py"
],
"testEnvironment": false,
"analysisUpdates": true,
"traceLogging": true,
"asyncStartup": true
},
"trace": "verbose",
"workspaceFolders": [
{
"uri": "file:///Users/malmaud/dropbox/code/qa",
"name": "qa"
}
]
}
[Trace - 3:43:39 PM] Received notification 'window/logMessage'.
Params: {
"type": 3,
"message": "Microsoft Python Language Server version 0.4.67.0"
}
[Info - 3:43:39 PM] Microsoft Python Language Server version 0.4.67.0
[Trace - 3:43:39 PM] Received notification 'window/logMessage'.
Params: {
"type": 3,
"message": "Workspace root: /Users/malmaud/dropbox/code/qa"
}
[Info - 3:43:39 PM] Workspace root: /Users/malmaud/dropbox/code/qa
[Trace - 3:43:39 PM] Received response 'initialize - (0)' in 877ms.
Result: {
"capabilities": {
"textDocumentSync": {
"openClose": true,
"change": 2,
"willSave": false,
"willSaveWaitUntil": false
},
"hoverProvider": true,
"completionProvider": {
"resolveProvider": false,
"triggerCharacters": [
"."
]
},
"signatureHelpProvider": {
"triggerCharacters": [
"(",
",",
")"
]
},
"definitionProvider": true,
"referencesProvider": true,
"documentHighlightProvider": false,
"documentSymbolProvider": true,
"workspaceSymbolProvider": true,
"codeActionProvider": false,
"documentFormattingProvider": false,
"documentRangeFormattingProvider": false,
"documentOnTypeFormattingProvider": {
"firstTriggerCharacter": "\n",
"moreTriggerCharacter": [
";",
":"
]
},
"renameProvider": true,
"declarationProvider": true
}
}
[Trace - 3:43:39 PM] Sending notification 'initialized'.
Params: {}
[Trace - 3:43:39 PM] Sending notification 'workspace/didChangeConfiguration'.
Params: {
"settings": {
"python": {
"diagnostics": {
"sourceMapsEnabled": false
},
"autoComplete": {
"addBrackets": false,
"extraPaths": [],
"showAdvancedMembers": true,
"typeshedPaths": []
},
"autoUpdateLanguageServer": true,
"experiments": {
"enabled": true
},
"dataScience": {
"allowImportFromNotebook": true,
"gatherRules": [
{
"objectName": "df",
"functionName": "head",
"doesNotModify": [
"OBJECT"
]
},
{
"objectName": "df",
"functionName": "describe",
"doesNotModify": [
"OBJECT"
]
},
{
"objectName": "df",
"functionName": "tail",
"doesNotModify": [
"OBJECT"
]
},
{
"functionName": "print",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "KMeans",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "scatter",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "fit",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "sum",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "len",
"doesNotModify": [
"ARGUMENTS"
]
}
],
"askForLargeDataFrames": true,
"askForKernelRestart": true,
"enabled": true,
"exportWithOutputEnabled": false,
"jupyterLaunchTimeout": 60000,
"jupyterLaunchRetries": 3,
"jupyterServerURI": "local",
"notebookFileRoot": "${workspaceFolder}",
"searchForJupyter": true,
"changeDirOnImportExport": true,
"useDefaultConfigForJupyter": true,
"jupyterInterruptTimeout": 10000,
"allowInput": true,
"showCellInputCode": true,
"collapseCellInputCodeByDefault": true,
"maxOutputSize": 400,
"errorBackgroundColor": "#FFFFFF",
"sendSelectionToInteractiveWindow": true,
"showJupyterVariableExplorer": true,
"variableExplorerExclude": "module;function;builtin_function_or_method",
"codeRegularExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
"markdownRegularExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)",
"allowLiveShare": true,
"ignoreVscodeTheme": false,
"liveShareConnectionTimeout": 1000,
"decorateCells": true,
"enableCellCodeLens": true,
"enableAutoMoveToNextCell": true,
"autoPreviewNotebooksInInteractivePane": false,
"useNotebookEditor": true,
"allowUnauthorizedRemoteConnection": false,
"enablePlotViewer": true,
"enableGather": false,
"codeLenses": "python.datascience.runcell, python.datascience.runallcellsabove, python.datascience.debugcell",
"debugCodeLenses": "python.datascience.debugcontinue, python.datascience.debugstop, python.datascience.debugstepover",
"ptvsdDistPath": "",
"stopOnFirstLineWhileDebugging": true,
"remoteDebuggerPort": -1,
"textOutputLimit": 20000,
"colorizeInputBox": true,
"stopOnError": true,
"addGotoCodeLenses": true,
"magicCommandsAsComments": false,
"runMagicCommands": "",
"runStartupCommands": "",
"debugJustMyCode": true
},
"disableInstallationCheck": false,
"envFile": "${workspaceFolder}/.env",
"formatting": {
"autopep8Args": [],
"autopep8Path": "autopep8",
"provider": "black",
"blackArgs": [],
"blackPath": "black",
"yapfArgs": [],
"yapfPath": "yapf"
},
"globalModuleInstallation": false,
"jediEnabled": false,
"jediMemoryLimit": 0,
"jediPath": "",
"analysis": {
"openFilesOnly": true,
"diagnosticPublishDelay": 1000,
"errors": [],
"warnings": [],
"information": [],
"disabled": [],
"typeshedPaths": [],
"cacheFolderPath": "",
"memory": {
"keepLibraryAst": false,
"keepLibraryLocalVariables": false
},
"logLevel": "Trace",
"symbolsHierarchyDepthLimit": 10,
"cachingLevel": "Default",
"downloadChannel": "daily"
},
"linting": {
"enabled": true,
"flake8Args": [
"--select=F821,F706,F705,F822"
],
"flake8CategorySeverity": {
"E": "Error",
"F": "Error",
"W": "Warning"
},
"flake8Enabled": false,
"flake8Path": "flake8",
"ignorePatterns": [
".vscode/*.py",
"**/site-packages/**/*.py"
],
"lintOnSave": true,
"maxNumberOfProblems": 100,
"banditArgs": [],
"banditEnabled": false,
"banditPath": "bandit",
"mypyArgs": [
"--ignore-missing-imports",
"--follow-imports=silent",
"--show-column-numbers"
],
"mypyCategorySeverity": {
"error": "Error",
"note": "Information"
},
"mypyEnabled": true,
"mypyPath": "mypy",
"pycodestyleArgs": [],
"pycodestyleCategorySeverity": {
"E": "Error",
"W": "Warning"
},
"pycodestyleEnabled": false,
"pycodestylePath": "pycodestyle",
"prospectorArgs": [],
"prospectorEnabled": false,
"prospectorPath": "prospector",
"pydocstyleArgs": [],
"pydocstyleEnabled": false,
"pydocstylePath": "pydocstyle",
"pylamaArgs": [],
"pylamaEnabled": false,
"pylamaPath": "pylama",
"pylintArgs": [],
"pylintCategorySeverity": {
"convention": "Information",
"error": "Error",
"fatal": "Error",
"refactor": "Hint",
"warning": "Warning"
},
"pylintEnabled": false,
"pylintPath": "pylint",
"pylintUseMinimalCheckers": true
},
"pythonPath": "/Users/malmaud/anaconda3/bin/python",
"condaPath": "",
"pipenvPath": "pipenv",
"poetryPath": "poetry",
"sortImports": {
"args": [],
"path": ""
},
"terminal": {
"activateEnvironment": false,
"executeInFileDir": false,
"launchArgs": []
},
"testing": {
"cwd": null,
"debugPort": 3000,
"nosetestArgs": [],
"nosetestsEnabled": false,
"nosetestPath": "nosetests",
"promptToConfigure": false,
"pytestArgs": [],
"pytestEnabled": false,
"pytestPath": "pytest",
"unittestArgs": [
"-v",
"-s",
".",
"-p",
"*test*.py"
],
"unittestEnabled": false,
"autoTestDiscoverOnSaveEnabled": true
},
"venvFolders": [],
"venvPath": "",
"workspaceSymbols": {
"ctagsPath": "ctags",
"enabled": true,
"exclusionPatterns": [
"**/site-packages/**"
],
"rebuildOnFileSave": true,
"rebuildOnStart": true,
"tagFilePath": "${workspaceFolder}/.vscode/tags"
},
"insidersChannel": "off",
"trace": {
"server": "verbose"
}
}
}
}
[Trace - 3:43:39 PM] Sending notification 'textDocument/didOpen'.
Params: {
"textDocument": {
"uri": "file:///Users/malmaud/dropbox/code/qa/qa/load_data.py",
"languageId": "python",
"version": 1,
"text": "import os\nimport yaml\nimport pickle\nimport re\nfrom dataclasses import dataclass, field\nfrom typing import List, Sequence, Any, Optional, Generic, TypeVar, Tuple, Dict\nimport json\nfrom . import onestop_reader\nimport random\nfrom pathlib import Path\nfrom sklearn.model_selection import KFold\nimport numpy as np\nimport pandas as pd\nimport pytorch_transformers\n\n\nclass ContextID:\n pass\n\n\n@dataclass(unsafe_hash=True)\nclass OnestopID(ContextID):\n article_id: int\n article_name: str\n paragraph_id: int\n level: str\n\n\n@dataclass(unsafe_hash=True)\nclass RaceID(ContextID):\n partition: str = \"\"\n level: str = \"\"\n paragraph_id: int = -1\n json_id: str = \"\"\n\n\nT_context = TypeVar(\"T_context\", bound=ContextID)\nSpan_T = Optional[List[Tuple[int, int]]]\n\n\n@dataclass\nclass Document(Generic[T_context]):\n answers: List[int]\n options: List[List[str]]\n questions: List[str]\n article: str\n id: T_context\n spans: Optional[List[List[Span_T]]] = None\n\n\n@dataclass(unsafe_hash=True)\nclass QuestionID(Generic[T_context]):\n context_id: T_context\n question: int\n\n\nclass Example(Generic[T_context]):\n \"\"\"A single multiple-choice question\"\"\"\n\n id: QuestionID[T_context]\n context_sentence: str\n start_ending: str\n endings: List[str]\n label: Optional[int]\n a_span: Span_T\n d_span: Span_T\n gaze: Optional[List[float]]\n\n def __init__(\n self,\n id: QuestionID[T_context],\n context_sentence: str,\n start_ending: str,\n ending_0: str,\n ending_1: str,\n ending_2: str,\n ending_3: str,\n label: Optional[int] = None,\n gaze=None,\n ):\n self.id = id\n self.context_sentence = context_sentence\n self.start_ending = start_ending\n self.endings = [ending_0, ending_1, ending_2, ending_3]\n self.label = label\n self.gaze = gaze\n\n def get_id(self) -> int:\n if os.environ.get(\"PYTHONHASHSEED\", \"random\") != \"0\":\n print(\"Warning: PYTHONHASHSEED not static\")\n return hash(self.id)\n\n def __str__(self):\n return self.__repr__()\n\n def __repr__(self):\n l = [\n \"id: {}\".format(self.id),\n \"context_sentence: {}\".format(self.context_sentence),\n \"start_ending: {}\".format(self.start_ending),\n \"ending_0: {}\".format(self.endings[0]),\n \"ending_1: {}\".format(self.endings[1]),\n \"ending_2: {}\".format(self.endings[2]),\n \"ending_3: {}\".format(self.endings[3]),\n ]\n\n if self.label is not None:\n l.append(\"label: {}\".format(self.label))\n\n return \", \".join(l)\n\n\ndef parse_race_dir(train_dir) -> List[Example]:\n docs = []\n answer_map = {\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3}\n for path in train_dir.iterdir():\n parsed = json.loads(path.read_bytes())\n answers = [answer_map[_] for _ in parsed[\"answers\"]]\n m = re.match(r\".*?(\\d+)\\.txt$\", parsed[\"id\"])\n if (m is None) or len(m.groups()) < 1:\n raise Exception(f\"Malformed example {path}\")\n paragraph_id = int(m[1])\n example = Document(\n answers=answers,\n options=parsed[\"options\"],\n questions=parsed[\"questions\"],\n article=parsed[\"article\"],\n id=RaceID(json_id=parsed[\"id\"], paragraph_id=paragraph_id),\n )\n docs.append(example)\n examples = split_documents(docs)\n return examples\n\n\ndef parse_race(race_dir: Path, out_path: Optional[Path] = None):\n sets = [\"dev\", \"test\", \"train\"]\n levels = [\"high\", \"middle\"]\n all_examples: List[Example] = []\n for set_name in sets:\n for level in levels:\n examples = parse_race_dir(race_dir / set_name / level)\n for example in examples:\n example.id.context_id.partition = set_name\n example.id.context_id.level = level\n all_examples.extend(examples)\n if out_path is not None:\n pickle.dump(all_examples, out_path.open(\"wb\"))\n return all_examples\n\n\ndef load_race(\n force_parse: bool = False,\n level: Optional[str] = None,\n partition: Optional[str] = None,\n):\n data_dir = Path(Path(__file__).parent.parent / \"data\")\n race_dir = data_dir / \"RACE\"\n out_file = data_dir / \"RACE.pickle\"\n if force_parse or not out_file.is_file():\n parse_race(race_dir, out_file)\n examples: List[Example] = pickle.load(out_file.open(\"rb\"))\n if level is not None:\n examples = [\n example for example in examples if example.id.context_id.level == level\n ]\n if partition is not None:\n examples = [\n example\n for example in examples\n if example.id.context_id.partition == partition\n ]\n return examples\n\n\ndef split_documents(docs: Sequence[Document]) -> List[Example]:\n examples = []\n for doc in docs:\n n_questions = len(doc.questions)\n for i in range(n_questions):\n example = Example(\n id=QuestionID(context_id=doc.id, question=i),\n context_sentence=doc.article,\n start_ending=doc.questions[i],\n ending_0=doc.options[i][0],\n ending_1=doc.options[i][1],\n ending_2=doc.options[i][2],\n ending_3=doc.options[i][3],\n label=doc.answers[i],\n )\n if doc.spans is not None:\n a_span, d_span = doc.spans\n example.a_span = a_span[i]\n example.d_span = d_span[i]\n examples.append(example)\n return examples\n\n\ndef get_onestop_dir() -> Path:\n # The redundant `Path` here is because of a bug with VSCode's analysis\n return Path(Path(__file__).parent.parent / \"data\") / \"onestop_articles\"\n\n\n@dataclass\nclass OnestopMetadata:\n batches: List[List[str]]\n practice: List[List[str]]\n train: List[str]\n test: List[str]\n\n\ndef load_onestop_metadata() -> OnestopMetadata:\n path = Path(__file__).parent.parent / \"data/onestop_metadata.yaml\"\n res = yaml.safe_load(path.open(\"rb\"))\n return OnestopMetadata(\n batches=res[\"batches\"],\n train=res[\"train\"],\n test=res[\"test\"],\n practice=res[\"practice\"],\n )\n\n\ndef filter_for_example(example: Example, df: pd.DataFrame) -> pd.DataFrame:\n context = example.id.context_id\n df = df[\n (df.file_name == context.article_name)\n & (example.id.question == df.q_ind)\n & (context.paragraph_id == df.paragraph_id)\n & (df.level == context.level)\n ]\n df = df.sort_values(\"IA_ID\")\n\n return df\n\n\ndef get_split(articles, partition, fold, max_folds=4):\n rng_state = np.random.get_state()\n np.random.seed(0)\n cars = []\n N = len(articles)\n per_fold = N // max_folds\n for i in range(4):\n cars.extend([i] * 5)\n np.random.shuffle(cars)\n train_cars = (np.array([0, 1]) + fold) % 4\n dev_cars = (np.array([2]) + fold) % 4\n test_cars = (np.array([3]) + fold) % 4\n partition_cars = {\"train\": train_cars, \"dev\": dev_cars, \"test\": test_cars}[\n partition\n ]\n res_articles = []\n for car in partition_cars:\n for article_idx, article_car in enumerate(cars):\n if article_car == car:\n res_articles.append(articles[article_idx])\n np.random.set_state(rng_state)\n return res_articles\n\n\ndef load_onestop(\n batch: Optional[int] = None,\n level=None,\n partition=None,\n fold=None,\n include_practice=False,\n gaze_df: pd.DataFrame = None,\n):\n onestop_articles = get_onestop_dir()\n articles = onestop_reader.read_all_articles(str(onestop_articles) + \"/\")\n docs = []\n for article in articles:\n for annotation in article.paragraph_annotations:\n for paragraph in annotation.paragraph_versions:\n questions = []\n options = []\n for question in annotation.questions:\n answers = []\n questions.append(question.question)\n for answer in question.answers:\n answers.append(answer)\n options.append(answers)\n onestop_id = OnestopID(\n article_id=article.article_id,\n article_name=article.article_name,\n paragraph_id=annotation.paragraph_id,\n level=paragraph.level,\n )\n doc = Document(\n answers=[0] * 4,\n options=options,\n questions=questions,\n article=paragraph.plain_text,\n id=onestop_id,\n spans=[paragraph.A_inds, paragraph.D_inds],\n )\n docs.append(doc)\n examples = split_documents(docs)\n metadata = load_onestop_metadata()\n if batch is not None:\n articles = [name + \".txt\" for name in metadata.batches[batch]]\n if include_practice:\n for name in metadata.practice[batch]:\n articles.append(name + \".txt\")\n examples = [\n example\n for example in examples\n if example.id.context_id.article_name in articles\n ]\n\n if level is not None:\n examples = [\n example for example in examples if example.id.context_id.level == level\n ]\n if partition is not None and fold is None:\n if partition == \"train\":\n article_set = metadata.train\n elif partition == \"test\":\n article_set = metadata.test\n article_set = [name + \".txt\" for name in article_set]\n examples = [\n example\n for example in examples\n if example.id.context_id.article_name in article_set\n ]\n if fold is not None:\n if partition is None:\n raise Exception(\n f\"Partition cannot be none when fold is none. Fold is {fold}\"\n )\n articles = np.unique(\n [example.id.context_id.article_name for example in examples]\n )\n split_articles = get_split(articles, partition, fold)\n\n examples = [\n example\n for example in examples\n if example.id.context_id.article_name in split_articles\n ]\n\n for example in examples:\n # There were some change markers like '#a' in the data\n example.start_ending = re.sub(r\"\\s+#.*$\", \"\", example.start_ending)\n if gaze_df is not None:\n examples = [\n example for example in examples if example.id.context_id.level != \"Int\"\n ]\n for example in examples:\n f = filter_for_example(example, gaze_df)\n example.gaze = f.dwell_time_normed.tolist()\n examples = [\n example\n for example in examples\n if (example.gaze is not None) and len(example.gaze) > 0\n ]\n return examples\n\n\n@dataclass\nclass AllData:\n race_m: Any = None\n race_h: Any = None\n race_all: Any = None\n onestop_ele: Any = None\n onestop_int: Any = None\n onestop_adv: Any = None\n onestop_all: Any = None\n\n @staticmethod\n def load(partition: str = \"test\"):\n return AllData(\n race_m=load_race(partition=partition, level=\"middle\"),\n race_h=load_race(partition=partition, level=\"high\"),\n race_all=load_race(partition=partition),\n onestop_ele=load_onestop(partition=partition, level=\"Ele\"),\n onestop_int=load_onestop(partition=partition, level=\"Int\"),\n onestop_adv=load_onestop(partition=partition, level=\"Adv\"),\n onestop_all=load_onestop(partition=partition),\n )\n\n\n@dataclass\nclass RaceData:\n race_m: Any = None\n race_h: Any = None\n race_all: Any = None\n\n @staticmethod\n def load(partition: str = \"test\"):\n return RaceData(\n race_m=load_race(partition=partition, level=\"middle\"),\n race_h=load_race(partition=partition, level=\"high\"),\n race_all=load_race(partition=partition),\n )\n\n\n@dataclass\nclass OnestopData:\n\n onestop_ele: Any = None\n onestop_int: Any = None\n onestop_adv: Any = None\n onestop_all: Any = None\n\n @staticmethod\n def load(partition: Optional[str] = \"test\", fold=None):\n return OnestopData(\n onestop_ele=load_onestop(partition=partition, level=\"Ele\", fold=fold),\n onestop_int=load_onestop(partition=partition, level=\"Int\", fold=fold),\n onestop_adv=load_onestop(partition=partition, level=\"Adv\", fold=fold),\n onestop_all=load_onestop(partition=partition, fold=fold),\n )\n\n\ndef load_dataset(name, **kwargs):\n kind = name.split(\"_\")[0]\n if kind == \"race\":\n return RaceData.load(**kwargs)\n elif kind == \"onestop\":\n return OnestopData.load(**kwargs)\n else:\n raise Exception(\"Unrecognized kind\")\n\n\ndef examples_to_df(examples: List[Example]) -> pd.DataFrame:\n rows = []\n for e in examples:\n rows.append(\n {\n \"article_name\": e.id.context_id.article_name,\n \"article_id\": e.id.context_id.article_id,\n \"paragraph_id\": e.id.context_id.paragraph_id,\n \"level\": e.id.context_id.level,\n \"question_id\": e.id.question,\n \"context\": e.context_sentence,\n \"question\": e.start_ending,\n \"a\": e.endings[0],\n \"b\": e.endings[1],\n \"c\": e.endings[2],\n \"d\": e.endings[3],\n }\n )\n df = pd.DataFrame(rows)\n return df\n\n\ndef examples_by_hash() -> Dict[int, Example[OnestopID]]:\n example_hash = {}\n for example in load_onestop():\n example_hash[example.get_id()] = example\n return example_hash\n\n\ndef onestop_spreadsheet(save=False) -> pd.DataFrame:\n onestop = load_onestop()\n df = examples_to_df(onestop)\n df[\"hash\"] = [example.get_id() for example in onestop]\n batches = load_onestop_metadata().batches\n\n def get_batch(row):\n for batch_idx, batch in enumerate(batches):\n for article in batch:\n if (article + \".txt\") == row.article_name:\n return batch_idx + 1\n\n df[\"batch\"] = df.apply(get_batch, axis=1)\n if save:\n df.to_csv(\"examples.csv\")\n return df\n"
}
}
[Trace - 3:43:39 PM] Received request 'workspace/configuration - (2)'.
Params: {
"items": [
{
"section": "python"
}
]
}
[Trace - 3:43:39 PM] Sending response 'workspace/configuration - (2)'. Processing request took 6ms
Result: [
{
"diagnostics": {
"sourceMapsEnabled": false
},
"autoComplete": {
"addBrackets": false,
"extraPaths": [],
"showAdvancedMembers": true,
"typeshedPaths": []
},
"autoUpdateLanguageServer": true,
"experiments": {
"enabled": true
},
"dataScience": {
"allowImportFromNotebook": true,
"gatherRules": [
{
"objectName": "df",
"functionName": "head",
"doesNotModify": [
"OBJECT"
]
},
{
"objectName": "df",
"functionName": "describe",
"doesNotModify": [
"OBJECT"
]
},
{
"objectName": "df",
"functionName": "tail",
"doesNotModify": [
"OBJECT"
]
},
{
"functionName": "print",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "KMeans",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "scatter",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "fit",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "sum",
"doesNotModify": [
"ARGUMENTS"
]
},
{
"functionName": "len",
"doesNotModify": [
"ARGUMENTS"
]
}
],
"askForLargeDataFrames": true,
"askForKernelRestart": true,
"enabled": true,
"exportWithOutputEnabled": false,
"jupyterLaunchTimeout": 60000,
"jupyterLaunchRetries": 3,
"jupyterServerURI": "local",
"notebookFileRoot": "${workspaceFolder}",
"searchForJupyter": true,
"changeDirOnImportExport": true,
"useDefaultConfigForJupyter": true,
"jupyterInterruptTimeout": 10000,
"allowInput": true,
"showCellInputCode": true,
"collapseCellInputCodeByDefault": true,
"maxOutputSize": 400,
"errorBackgroundColor": "#FFFFFF",
"sendSelectionToInteractiveWindow": true,
"showJupyterVariableExplorer": true,
"variableExplorerExclude": "module;function;builtin_function_or_method",
"codeRegularExpression": "^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",
"markdownRegularExpression": "^(#\\s*%%\\s*\\[markdown\\]|#\\s*\\<markdowncell\\>)",
"allowLiveShare": true,
"ignoreVscodeTheme": false,
"liveShareConnectionTimeout": 1000,
"decorateCells": true,
"enableCellCodeLens": true,
"enableAutoMoveToNextCell": true,
"autoPreviewNotebooksInInteractivePane": false,
"useNotebookEditor": true,
"allowUnauthorizedRemoteConnection": false,
"enablePlotViewer": true,
"enableGather": false,
"codeLenses": "python.datascience.runcell, python.datascience.runallcellsabove, python.datascience.debugcell",
"debugCodeLenses": "python.datascience.debugcontinue, python.datascience.debugstop, python.datascience.debugstepover",
"ptvsdDistPath": "",
"stopOnFirstLineWhileDebugging": true,
"remoteDebuggerPort": -1,
"textOutputLimit": 20000,
"colorizeInputBox": true,
"stopOnError": true,
"addGotoCodeLenses": true,
"magicCommandsAsComments": false,
"runMagicCommands": "",
"runStartupCommands": "",
"debugJustMyCode": true
},
"disableInstallationCheck": false,
"envFile": "${workspaceFolder}/.env",
"formatting": {
"autopep8Args": [],
"autopep8Path": "autopep8",
"provider": "black",
"blackArgs": [],
"blackPath": "black",
"yapfArgs": [],
"yapfPath": "yapf"
},
"globalModuleInstallation": false,
"jediEnabled": false,
"jediMemoryLimit": 0,
"jediPath": "",
"analysis": {
"openFilesOnly": true,
"diagnosticPublishDelay": 1000,
"errors": [],
"warnings": [],
"information": [],
"disabled": [],
"typeshedPaths": [],
"cacheFolderPath": "",
"memory": {
"keepLibraryAst": false,
"keepLibraryLocalVariables": false
},
"logLevel": "Trace",
"symbolsHierarchyDepthLimit": 10,
"cachingLevel": "Default",
"downloadChannel": "daily"
},
"linting": {
"enabled": true,
"flake8Args": [
"--select=F821,F706,F705,F822"
],
"flake8CategorySeverity": {
"E": "Error",
"F": "Error",
"W": "Warning"
},
"flake8Enabled": false,
"flake8Path": "flake8",
"ignorePatterns": [
".vscode/*.py",
"**/site-packages/**/*.py"
],
"lintOnSave": true,
"maxNumberOfProblems": 100,
"banditArgs": [],
"banditEnabled": false,
"banditPath": "bandit",
"mypyArgs": [
"--ignore-missing-imports",
"--follow-imports=silent",
"--show-column-numbers"
],
"mypyCategorySeverity": {
"error": "Error",
"note": "Information"
},
"mypyEnabled": true,
"mypyPath": "mypy",
"pycodestyleArgs": [],
"pycodestyleCategorySeverity": {
"E": "Error",
"W": "Warning"
},
"pycodestyleEnabled": false,
"pycodestylePath": "pycodestyle",
"prospectorArgs": [],
"prospectorEnabled": false,
"prospectorPath": "prospector",
"pydocstyleArgs": [],
"pydocstyleEnabled": false,
"pydocstylePath": "pydocstyle",
"pylamaArgs": [],
"pylamaEnabled": false,
"pylamaPath": "pylama",
"pylintArgs": [],
"pylintCategorySeverity": {
"convention": "Information",
"error": "Error",
"fatal": "Error",
"refactor": "Hint",
"warning": "Warning"
},
"pylintEnabled": false,
"pylintPath": "pylint",
"pylintUseMinimalCheckers": true
},
"pythonPath": "/Users/malmaud/anaconda3/bin/python",
"condaPath": "",
"pipenvPath": "pipenv",
"poetryPath": "poetry",
"sortImports": {
"args": [],
"path": ""
},
"terminal": {
"activateEnvironment": false,
"executeInFileDir": false,
"launchArgs": []
},
"testing": {
"cwd": null,
"debugPort": 3000,
"nosetestArgs": [],
"nosetestsEnabled": false,
"nosetestPath": "nosetests",
"promptToConfigure": false,
"pytestArgs": [],
"pytestEnabled": false,
"pytestPath": "pytest",
"unittestArgs": [
"-v",
"-s",
".",
"-p",
"*test*.py"
],
"unittestEnabled": false,
"autoTestDiscoverOnSaveEnabled": true
},
"venvFolders": [],
"venvPath": "",
"workspaceSymbols": {
"ctagsPath": "ctags",
"enabled": true,
"exclusionPatterns": [
"**/site-packages/**"
],
"rebuildOnFileSave": true,
"rebuildOnStart": true,
"tagFilePath": "${workspaceFolder}/.vscode/tags"
},
"insidersChannel": "off",
"trace": {
"server": "verbose"
}
}
]
I only see 2 which are normal - unless DatabasePath which is technically obsolete, leaks into the indexer.
"interpreter": {
"properties": {
"InterpreterPath": "/Users/malmaud/anaconda3/bin/python",
"Version": "3.7.2",
"DatabasePath": "/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67"
}
},
"searchPaths": [],
"typeStubSearchPaths": [
"/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67/Typeshed"
],Great, thank you. The only references to the language server folder are the typeshed search path, and the database path (unused). No search paths are set any other way in the configuration I'm seeing here, which means the extension isn't setting anything interesting and the issue is later, maybe at get_search_paths.py (whose directory is the LS folder).
Can you use VS Code to open a terminal with the environment activated, then run:
$ python /Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67/get_search_paths.pyAnd show the output?
malmaud
commented
5 years ago File "/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67/get_search_paths.py", line 27, in <module>
raise RuntimeError('script must be run with -S')
RuntimeError: script must be run with -SBah, sorry. Do python -S -E <that long path>.
malmaud
commented
5 years ago /Users/malmaud/anaconda3/lib/python3.7/site-packages/_pdbpp_path_hack|pth|
/Users/malmaud/.vscode/extensions/ms-python.python-2019.10.41019/languageServer.0.4.67|stdlib|
/Users/malmaud/anaconda3/lib/python3.7|stdlib|
/Users/malmaud/anaconda3/lib/python3.7/lib-dynload|stdlib|
/Users/malmaud/anaconda3/lib/python3.7/site-packages|site|
/Users/malmaud/anaconda3/lib/python3.7/site-packages/aeosa|pth|There's the problem. Something is managing to insert itself before the script's own path, so is probably some wacky pth file.
We'll probably need to undo the Skip(1) and simply ignore the path we run the script in when reading this info as we used to do a bit ago. (Or modify the get_search_path script to identify its own directory and omit it, which is probably safer.)
malmaud
commented
5 years ago I notice pdbpp in that output, which is a reference to https://pypi.org/project/pdbpp/. It might be doing some kind of import magic which is related.
jakebailey
commented
5 years ago For now you should be able to just remove that package to fix the issue, but we'll still have to fix it generally. I'm not sure how popular this package is, but if it's made its way into a typical anaconda environment that might explain the prevalence of this exception.
jakebailey
commented
5 years ago Will be available in 0.4.68+ (once the build finishes). I would appreciate it if you could confirm that things are fixed!
Seeing this in the output log: