Closed tcosz closed 2 months ago
dtzikas
commented
3 months ago A while ago I had created a pull request to import gds from istream (in order to support reading gzipped files). If you think it could be related to the "missing part" here, you can have a look at recently rebased branch
tcosz
commented
3 months ago Thank you for sharing your rebased branch, based on your feedback in the case of this thread a solution could be to use sstream (for std::ostringstream) which does not seem to add new external dependencies.
Eventually could the default implementation could also benefit from using the steam from the start when building up the the GDS data before writing to the file? In this approach all the procedures to export to GDS (cells, polygons, paths, etc) would need to replace the file input variable a the stream input variable. I am adding some documentation below for the discussion, could be interesting to run a new benchmark with a version of the codebase based on the stream.
Using std::ostream instead of FILE* for writing data can have several implications on performance and usability. Here are some points to consider:
BytesIOUsing std::ostream is better compatible with exporting data for later use with Python's BytesIO because it provides a more flexible and seamless way to handle in-memory data streams. Here’s why:
Seamless Data Handling:
std::ostream allows you to directly write data to an in-memory stream (like std::ostringstream). This data can then be easily converted to a standard container like std::vector<char>, which can be directly passed to Python as a bytes object.Direct Integration:
std::ostream, the written data can be captured directly into a string or byte array, facilitating straightforward integration with Python's BytesIO. This eliminates the need for intermediate file operations and reduces I/O overhead.Unified Interface:
std::ostream provides a unified interface for writing data, making the code cleaner and more modular. You can use the same function to write to files, network streams, or in-memory buffers, enhancing code reuse and maintainability.Buffered I/O:
std::ostream and FILE* typically use buffered I/O. The buffering mechanism helps in reducing the number of system calls, which improves performance when writing large amounts of data.std::ostream uses the C++ standard library's buffering, while FILE* uses the C standard library's buffering. The performance difference between these two might be negligible for most applications, but it could vary based on the implementation and the specific usage scenario.Flexibility and Abstraction:
std::ostream provides a higher level of abstraction, allowing you to write to various types of streams (e.g., file streams, string streams, custom streams) with the same interface.Inline Function Calls:
std::ostream, there is often less need for explicit memory management and error handling within the I/O code itself. This can lead to cleaner and potentially faster code due to fewer function calls and less error-checking overhead.Stream Operators:
<< and >>) in std::ostream can sometimes be more efficient for formatting and writing data compared to fprintf and fwrite with FILE*.Error Handling:
std::ostream provides better error handling through its state flags (e.g., badbit, failbit). This can simplify error detection and handling in your code.FILE*, you often need to check return values and handle errors explicitly.Code Clarity:
std::ostream can make the code more readable and expressive. The stream interface is often more intuitive and easier to use, especially for C++ developers familiar with the standard library.Integration with C++ Features:
std::ostream integrates seamlessly with other C++ features like RAII (Resource Acquisition Is Initialization), smart pointers, and the standard library's algorithms and containers.For a concrete performance comparison, you would typically need to measure the actual performance of both implementations in your specific use case. Here's a simplified example to illustrate potential differences:
FILE*#include <cstdio>
#include <chrono>
void write_using_file(const char* filename, const char* data, size_t size) {
FILE* file = fopen(filename, "wb");
if (file) {
fwrite(data, 1, size, file);
fclose(file);
}
}
int main() {
const size_t size = 1024 * 1024 * 100; // 100 MB
char* data = new char[size];
std::fill_n(data, size, 'A');
auto start = std::chrono::high_resolution_clock::now();
write_using_file("output_file.bin", data, size);
auto end = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> elapsed = end - start;
printf("FILE* write time: %.6f seconds\n", elapsed.count());
delete[] data;
return 0;
}std::ostream#include <fstream>
#include <chrono>
void write_using_stream(const char* filename, const char* data, size_t size) {
std::ofstream file(filename, std::ios::binary);
if (file) {
file.write(data, size);
}
}
int main() {
const size_t size = 1024 * 1024 * 100; // 100 MB
char* data = new char[size];
std::fill_n(data, size, 'A');
auto start = std::chrono::high_resolution_clock::now();
write_using_stream("output_stream.bin", data, size);
auto end = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> elapsed = end - start;
printf("std::ostream write time: %.6f seconds\n", elapsed.count());
delete[] data;
return 0;
}std::ostream and FILE* is typically small, but it can vary based on the implementation and specific use case. For most applications, the difference is negligible.std::ostream offers better usability, maintainability, and integration with C++ features, which can lead to safer and more robust code.std::ostream in handling different types of streams makes it a better choice for many C++ applications.For modern C++ development, std::ostream is generally preferred due to its integration with the standard library and better abstraction.
heitzmann
commented
3 months ago Unfortunately I can't take the required time to implement this modernization, but I'm now open to merging PRs with it (seeing as it will facilitate for future contributions too).
If anyone is willing to take ownership of the project, I'm open to it as well, because I haven't been able to keep up with feature requests for the past year. @nmz787-intel @tvt173 @wshanks @jatoben @dtzikas @tcosz @joamatab
tvt173
commented
2 months ago Thanks @heitzmann. gdspy and gdstk have been invaluable to me over the past decade or so in various projects and as the original backends for gdsfactory. However, we recently switched the backend of gdsfactory to klayout, and I have no time to maintain another open source package either. I invite anyone interested in an alternative to give gdsfactory a try though!
nmz787-intel
commented
2 months ago FYI if you're on linux you can get away with just using a tmpfs mount point and no code updates to GDSTK:
import subprocess
import tempfile
import os
def check_ram_disk(mount_point='/dev/shm'):
try:
# Execute the mount command and decode the output
result = subprocess.run(['mount'], stdout=subprocess.PIPE)
output = result.stdout.decode('utf-8')
# Check if the specified mount point is mounted as tmpfs
return f'{mount_point} type tmpfs' in output
except Exception as e:
print(f"Error checking RAM disk: {e}")
return False
def create_temp_file_in_ram(mount_point='/dev/shm'):
if check_ram_disk(mount_point):
# Create a temporary file in the specified RAM disk
with tempfile.NamedTemporaryFile(dir=mount_point, delete=False) as tmp_file:
print(f'Temporary file created at: {tmp_file.name}')
# Write some data to the temporary file
tmp_file.write(b'Hello, this is a test data written to RAM disk!')
tmp_file.flush() # Ensure all data is written to the file
# Read and print the content from the file
with open(tmp_file.name, 'rb') as file:
content = file.read()
print(f'Content of the file: {content}')
# Clean up the file after use
os.unlink(tmp_file.name)
print(f'File {tmp_file.name} has been deleted.')
else:
print(f"{mount_point} is not mounted as tmpfs. RAM disk is not available.")Thank you very much for the tip for writing to the RAM directly from Python. Closing the thread since the C++ code update to implement IO streaming would require a lot of efforts.
Hello,
First of all thank you very much for providing such an powerful and easy to use package. We require to keep the GDS data in memory to pass it as input to another Python package function, currently the code only supports exporting to a file. We believe adding this functionality to buffer the output in-memory will solve the limitations we have to use
gdstkpackage for preprocessing complex layout data.Could you please consider adding the functionality in the core package? Here is a suggested code implementation. Unfortunately despite our best attempts we are not able to correctly install an editable package version with the newly compiled code, but we believe the effort to implement should be limited.
Typically this is what we would like to be able to do in Python
The main idea is to replace in a new
write_gds_to_bufferfunction the first argument of theLibrary::write_gdsfunction,const char* filename, bystd::vector<char>& buffer. and to replace the file output by (complete code towrite_gds_to_bufferis pasted further below).*include/gdstk/library.hpp** ( inside struct Library)
*src/library.cpp** (inside namespace gdstk)
#include <gdstk/vec.hpp>in the include sectionwrite_gds_to_bufferfunction*python/library_object.cpp**
Of course having a similar feature for the import_gds would be good for consistency but at the moment the enhancement on write_gds would help greatly.
Thank you.